You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/16 18:56:06 UTC
[GitHub] [pulsar] dlg99 opened a new pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
dlg99 opened a new pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927
### Motivation
Provide a way to use Kafka-Connect Sink as a Pulsar Sink, in cases like:
- company has custom kafka sink and want to try the pulsar out
- no corresponding pulsar sink exists
etc.
### Modifications
Added KafkaConnectSink. Kafka Schema is autodetected from key/value itself for primitive values with option to unwrap KeyValue. GenericRecord support is TBD, depends on the changes Enrico is working on.
Added profile to pulsar-io/kafka-connect-adaptor to build nar with kafka-connect connector included
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
Added unit tests.
Tested locally as
Ran pulsar standalone `bin/pulsar standalone`
Built nar as `mvn -f pulsar-io/kafka-connect-adaptor-nar/pom.xml clean package -DskipTests -P packageKafkaConnect` to include kafka's connect-file sink into the nar.
Ran test nar as
```
bin/pulsar-admin sinks localrun -a ./pulsar-io/kafka-connect-adaptor-nar/target/pulsar-io-kafka-connect-adaptor-nar-2.8.0-SNAPSHOT.nar --name kwrap --namespace public/default/ktest --parallelism 1 -i my-topic --sink-config-file ~/sink.yaml
```
with
```
$ cat ~/sink.yaml
configs:
"topic": "test"
"offsetStorageTopic": "kafka-connect-sink-offset"
"pulsarServiceUrl": "pulsar://localhost:6650/"
"kafkaConnectorSinkClass": "org.apache.kafka.connect.file.FileStreamSinkConnector"
"defaultKeySchema": "STRING_SCHEMA"
"defaultValueSchema": "BYTES_SCHEMA"
"kafkaConnectorConfigProperties":
"file": "/tmp/sink_test.out"
```
set topic schema as
```
bin/pulsar-admin schemas upload --filename ~/schema.json my-topic
$ cat ~/schema.json
{
"type": "STRING",
"schema": "",
"properties": {
"key1": "value1"
}
}
```
message produced as
```
bin/pulsar-client produce my-topic --messages "hello-pulsar"
```
and got
```
$ cat /tmp/sink_test.out
hello-pulsar
```
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not documented yet)
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
TBD following review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r597269231
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
+ KeyValue<Object, Object> kv = (KeyValue<Object, Object>) sourceRecord.getValue();
+ key = kv.getKey();
+ value = kv.getValue();
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue();
+ }
+ final Schema keySchema = getKafkaConnectSchemaForObject(key, defaultKeySchema);
+ final Schema valueSchema = getKafkaConnectSchemaForObject(value, defaultValueSchema);
+
+ long offset = taskContext.currentOffset(topicName, partition).incrementAndGet();
Review comment:
@sijie The Record<> may or may not contain sequenceID (Optional)
sequenceID, at the best, is `Optional.of(FunctionCommon.getSequenceId(message.getMessageId()))` - as i understand this applies to persistent topics only.
Record may or may not contain Message (Optional): sourceRecord.getMessage() which may return sequenceID or -1 (kafka has a thing against negative offsets, SinkRecord does not allow them).
MessageId can return byte[] (sourceRecord.getMessage().get().getMessageId().toByteArray()) of something - not guaranteed to be offset, rather a byte[] of anything, i.e. ledgerId, entryId, batchSize, topicName etc. Or it can throw UnsupportedOperationException
AFAICT usable sequenceID can be retrieved from data produced by kafka source (and there is absolutely no point in connecting kafka source to kafka sink in pulsar) or from the data read from bookkeeper.
I can do something along the lines of
```
long offset = sourceRecord.getRecordSequence()
.orElse(taskContext.currentOffset(topicName, partition).incrementAndGet());
```
This assumes that topic either has sequenceID for all records, or does not have them for all records.
Next problem is that kafka's offsets increase monotonically (per partition).
In case of BK, offsets is `(ledgerId << 28) | entryId` and is not guaranteed to be sequential, as I understand.
I don't know how kafka sinks react to that. Some sinks ignore offsets, others may track specific invariants.
Simply assigning the offsets the way I do now is the simplest solution.
It won't allow Exactly-once processing if the kafka sink deduping/overwriting data by offset id (and not the key, and if it supports this) but it will work as AtLeast-once.
I.e. if everything restarts it won't re-assign the same offsets (so deduping sink won't overwrite older data) but in case of replay/retry it will assign new offsets to the data potentially processed by the sink.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r621335621
##########
File path: pulsar-io/kafka-connect-adaptor-nar/pom.xml
##########
@@ -30,13 +30,40 @@
<artifactId>pulsar-io-kafka-connect-adaptor-nar</artifactId>
<name>Pulsar IO :: Kafka Connect Adaptor NAR</name>
- <dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
- <version>${project.version}</version>
- </dependency>
- </dependencies>
+ <profiles>
+ <profile>
+ <id>main</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
Review comment:
I am not sure it is a good idea to add an example to package a file connect here. Isn't it better to add a separate example module for demonstration purposes?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
Review comment:
two questions here:
1) why do we need this transformation here? Can't we just use `pendingFlushQueue.size()`?
2) why do we use link queue not other queue implementation which provides more efficient `size()` implementation?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkContext.java
##########
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.connect.connector.ConnectorContext;
+
+@Slf4j
+public class PulsarKafkaSinkContext implements ConnectorContext {
+
+ @Override
+ public void requestTaskReconfiguration() {
+ throw new UnsupportedOperationException("not implemented");
+ }
+
+ @Override
+ public void raiseError(Exception e) {
+ log.error("raiseError requested", e);
Review comment:
If it is not supported, why do you want to log this message?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ SinkContext ctx,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e);
+ } catch (ExecutionException e) {
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e); }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
+ log.warn("timeout() is called but is not supported currently.");
+ }
+
+ @Override
+ public Set<TopicPartition> assignment() {
+ return currentOffsets.keySet();
+ }
+
+ @Override
+ public void pause(TopicPartition... topicPartitions) {
+ log.warn("pause() is called but is not supported currently.");
+ }
+
+ @Override
+ public void resume(TopicPartition... topicPartitions) {
+ log.warn("resume() is called but is not supported currently.");
+ }
+
+ @Override
+ public void requestCommit() {
+ log.warn("requestCommit() is called but is not supported currently.");
+ }
+
+ public void flushOffsets(Map<TopicPartition, OffsetAndMetadata> offsets) throws Exception {
+ Map<ByteBuffer, ByteBuffer> offsetMap = Maps.newHashMapWithExpectedSize(offsets.size());
+
+ offsets.forEach((tp, om) -> fillOffsetMap(offsetMap, tp, om.offset()));
+ CompletableFuture<Void> result = new CompletableFuture();
+ offsetStore.set(offsetMap, (ex, ignore) -> {
+ if (ex == null) {
+ result.complete(null);
+ } else {
+ log.error("error flushing offsets for {}",
+ Joiner.on(",").withKeyValueSeparator("=").join(offsets), ex);
+ result.completeExceptionally(ex);
+ }
+ });
+ result.get();
Review comment:
does this need to be blocking?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj - Object to get schema of.
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null && pulsarSchema.getSchemaInfo() != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null || pulsarSchema.getSchemaInfo() == null
+ ? "null"
+ : pulsarSchema.getSchemaInfo().toString())
+ + " object class "
+ + (obj == null ? "null" : obj.getClass().getCanonicalName()));
+ }
+ return result;
+ }
+
+
+ @SuppressWarnings("rawtypes")
+ private SinkRecord toSinkRecord(Record<GenericObject> sourceRecord) {
+ final int partition = sourceRecord.getPartitionIndex().orElse(0);
+ final String topic = sourceRecord.getTopicName().orElse(topicName);
+ final Object key;
+ final Object value;
+ final Schema keySchema;
+ final Schema valueSchema;
+
+ // sourceRecord is never instanceof KVRecord
+ // https://github.com/apache/pulsar/pull/10113
+ if (unwrapKeyValueIfAvailable && sourceRecord.getSchema() != null
+ && sourceRecord.getSchema().getSchemaInfo() != null
+ && sourceRecord.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ KeyValueSchema kvSchema = (KeyValueSchema) sourceRecord.getSchema();
+ KeyValue kv = (KeyValue) sourceRecord.getValue().getNativeObject();
+ key = kv.getKey();
+ value = kv.getValue();
+ keySchema = getKafkaConnectSchema(kvSchema.getKeySchema(), key);
+ valueSchema = getKafkaConnectSchema(kvSchema.getValueSchema(), value);
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue().getNativeObject();
+ keySchema = Schema.STRING_SCHEMA;
+ valueSchema = getKafkaConnectSchema(sourceRecord.getSchema(), value);
+ }
+
+ long offset = sourceRecord.getRecordSequence()
+ .orElse(-1L);
+ if (offset < 0) {
+ offset = taskContext.currentOffset(topic, partition)
+ .incrementAndGet();
+ } else {
+ final long curr = offset;
+ taskContext.currentOffset(topic, partition)
+ .updateAndGet(curMax -> Math.max(curr, curMax));
+ }
+
+ Long timestamp = null;
+ TimestampType timestampType = TimestampType.NO_TIMESTAMP_TYPE;
+ if (sourceRecord.getEventTime().isPresent()) {
+ timestamp = sourceRecord.getEventTime().get();
+ timestampType = TimestampType.CREATE_TIME;
+ } else if (sourceRecord.getMessage().isPresent()) {
+ timestamp = sourceRecord.getMessage().get().getPublishTime();
+ timestampType = TimestampType.LOG_APPEND_TIME;
Review comment:
Technically publishTime is not log append time. I think we should call it out in the documentation.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ SinkContext ctx,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e);
+ } catch (ExecutionException e) {
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e); }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
+ log.warn("timeout() is called but is not supported currently.");
+ }
+
+ @Override
+ public Set<TopicPartition> assignment() {
+ return currentOffsets.keySet();
+ }
+
+ @Override
+ public void pause(TopicPartition... topicPartitions) {
+ log.warn("pause() is called but is not supported currently.");
+ }
+
+ @Override
+ public void resume(TopicPartition... topicPartitions) {
+ log.warn("resume() is called but is not supported currently.");
+ }
+
+ @Override
+ public void requestCommit() {
+ log.warn("requestCommit() is called but is not supported currently.");
+ }
+
+ public void flushOffsets(Map<TopicPartition, OffsetAndMetadata> offsets) throws Exception {
+ Map<ByteBuffer, ByteBuffer> offsetMap = Maps.newHashMapWithExpectedSize(offsets.size());
+
+ offsets.forEach((tp, om) -> fillOffsetMap(offsetMap, tp, om.offset()));
+ CompletableFuture<Void> result = new CompletableFuture();
+ offsetStore.set(offsetMap, (ex, ignore) -> {
+ if (ex == null) {
+ result.complete(null);
+ } else {
+ log.error("error flushing offsets for {}",
+ Joiner.on(",").withKeyValueSeparator("=").join(offsets), ex);
Review comment:
Do we need to run `Joiner` here?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj - Object to get schema of.
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null && pulsarSchema.getSchemaInfo() != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null || pulsarSchema.getSchemaInfo() == null
+ ? "null"
+ : pulsarSchema.getSchemaInfo().toString())
+ + " object class "
+ + (obj == null ? "null" : obj.getClass().getCanonicalName()));
+ }
+ return result;
+ }
+
+
+ @SuppressWarnings("rawtypes")
+ private SinkRecord toSinkRecord(Record<GenericObject> sourceRecord) {
+ final int partition = sourceRecord.getPartitionIndex().orElse(0);
+ final String topic = sourceRecord.getTopicName().orElse(topicName);
+ final Object key;
+ final Object value;
+ final Schema keySchema;
+ final Schema valueSchema;
+
+ // sourceRecord is never instanceof KVRecord
+ // https://github.com/apache/pulsar/pull/10113
+ if (unwrapKeyValueIfAvailable && sourceRecord.getSchema() != null
+ && sourceRecord.getSchema().getSchemaInfo() != null
+ && sourceRecord.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ KeyValueSchema kvSchema = (KeyValueSchema) sourceRecord.getSchema();
+ KeyValue kv = (KeyValue) sourceRecord.getValue().getNativeObject();
+ key = kv.getKey();
+ value = kv.getValue();
+ keySchema = getKafkaConnectSchema(kvSchema.getKeySchema(), key);
+ valueSchema = getKafkaConnectSchema(kvSchema.getValueSchema(), value);
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue().getNativeObject();
+ keySchema = Schema.STRING_SCHEMA;
Review comment:
This should be `BYTES`. Pulsar uses `string` for storing keys doesn't mean the schema type is `String`. Because bytes are encoded as strings there as well.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj - Object to get schema of.
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null && pulsarSchema.getSchemaInfo() != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null || pulsarSchema.getSchemaInfo() == null
+ ? "null"
+ : pulsarSchema.getSchemaInfo().toString())
+ + " object class "
+ + (obj == null ? "null" : obj.getClass().getCanonicalName()));
+ }
+ return result;
+ }
+
+
+ @SuppressWarnings("rawtypes")
+ private SinkRecord toSinkRecord(Record<GenericObject> sourceRecord) {
+ final int partition = sourceRecord.getPartitionIndex().orElse(0);
+ final String topic = sourceRecord.getTopicName().orElse(topicName);
+ final Object key;
+ final Object value;
+ final Schema keySchema;
+ final Schema valueSchema;
+
+ // sourceRecord is never instanceof KVRecord
+ // https://github.com/apache/pulsar/pull/10113
+ if (unwrapKeyValueIfAvailable && sourceRecord.getSchema() != null
+ && sourceRecord.getSchema().getSchemaInfo() != null
+ && sourceRecord.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ KeyValueSchema kvSchema = (KeyValueSchema) sourceRecord.getSchema();
+ KeyValue kv = (KeyValue) sourceRecord.getValue().getNativeObject();
+ key = kv.getKey();
+ value = kv.getValue();
+ keySchema = getKafkaConnectSchema(kvSchema.getKeySchema(), key);
+ valueSchema = getKafkaConnectSchema(kvSchema.getValueSchema(), value);
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue().getNativeObject();
+ keySchema = Schema.STRING_SCHEMA;
+ valueSchema = getKafkaConnectSchema(sourceRecord.getSchema(), value);
+ }
+
+ long offset = sourceRecord.getRecordSequence()
+ .orElse(-1L);
+ if (offset < 0) {
+ offset = taskContext.currentOffset(topic, partition)
Review comment:
I don't think this is a good idea. We should fail fast on this. Because the source record here is from a Pulsar topic, it should always have the sequence id.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private int batchSize = 1;
Review comment:
1) if `batchSize` is used for representing number of messages, it should be renamed to `batchedMessages` or something else. `size` should be used for representing the total bytes.
2) You can and probably should implement the flush logic based on bytes not messages. As you said, `messages` is not good for calculating the total amount of memory, but `bytes` can.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
Review comment:
Do we need this error message? I fail it can be`info` or `warning`.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
Review comment:
We shouldn't increment the sequence id here. Because the input here is a Pulsar topic which should *always* have a sequence id. See comment in KafkaConnectSink class.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
Review comment:
KafkaBytesSink is different because it doesn't wrap any existing Kafka Connect Sink. Kafka Connect Sink implementation assumes data coming from Kafka topics which data are consumed in partition order. Hence we need to make sure we are receiving messages using a Failover subscription which is consistent with Kafka's consumption model. This is important to make things correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-822641966
rebased on latest master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-800665584
> I am not much convinced about the final packaging, because if you want to run an existing Kafka Sink you have to build this module and add that Sink as dependency (this is the same work we are doing with Debezium source?).
Final packaging is tricky.
What I have is an option to get started quickly.
I cannot include all possible kafka connect modules - most of the people will need only small subset of them (that's even if this won't create dependency conflicts), others will need some custom connector not available publicly.
Passing a jar assumes the fat jar -> either shading (size of nar increases) or dependency conflicts.
Merging two nars is an option https://jslabonte.wordpress.com/2012/02/01/how-to-merge-nar-files/
Either way, another profile is not for production use (I can't imagine file sink being very useful) but to provide an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r643256387
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ @VisibleForTesting
+ PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private long maxBatchSize;
+ private final AtomicLong currentBatchSize = new AtomicLong(0L);
+
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.warn("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ // while sourceRecord.getMessage() is Optional<>
+ // it should always be present in Sink which gets instance of PulsarRecord
+ // let's avoid checks for .isPresent() in teh rest of the code
+ Preconditions.checkArgument(sourceRecord.getMessage().isPresent());
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ currentBatchSize.addAndGet(sourceRecord.getMessage().get().size());
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ Preconditions.checkArgument(ctx.getSubscriptionType() == SubscriptionType.Failover
+ || ctx.getSubscriptionType() == SubscriptionType.Exclusive,
+ "Source must run with Exclusive or Failover subscription type");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ maxBatchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || currentBatchSize.get() >= maxBatchSize) {
Review comment:
@dlg99 This will cause submitting multiple flush tasks, no? Because `currentBatchSize` will always be larger than `maxBatchSize` before the `flush` action is taken.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "16384",
+ help = "Size of messages in bytes the sink will attempt to batch messages together before flush.")
+ private int batchSize = 16384;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "Time interval in milliseconds the sink will attempt to batch messages together before flush.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic;
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic;
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL to use for the offset store.")
+ private String pulsarServiceUrl;
Review comment:
This field will potentially be removed after PulsarClient is exposed to the context. So you might want to document it.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ @VisibleForTesting
+ PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private long maxBatchSize;
+ private final AtomicLong currentBatchSize = new AtomicLong(0L);
+
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.warn("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ // while sourceRecord.getMessage() is Optional<>
+ // it should always be present in Sink which gets instance of PulsarRecord
+ // let's avoid checks for .isPresent() in teh rest of the code
+ Preconditions.checkArgument(sourceRecord.getMessage().isPresent());
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ currentBatchSize.addAndGet(sourceRecord.getMessage().get().size());
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ Preconditions.checkArgument(ctx.getSubscriptionType() == SubscriptionType.Failover
+ || ctx.getSubscriptionType() == SubscriptionType.Exclusive,
+ "Source must run with Exclusive or Failover subscription type");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ maxBatchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || currentBatchSize.get() >= maxBatchSize) {
Review comment:
This potentially will enqueue a lot of tasks in the flush queue which will burn CPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-805384033
updated to reflect CR comments (except some that affect pulsar-io/context and can be done separately/later)
rebased on top of https://github.com/apache/pulsar/pull/9947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617663301
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
Review comment:
@sijie KVRecord is not handled by Sinks, it is used only for Sources.
see https://github.com/apache/pulsar/pull/10113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-827697390
@sijie can you please review ? this patch has been hanging for many days.
@merlimat @rdhabalia @codelipenghui can you help with merging this patch ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r597269231
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
+ KeyValue<Object, Object> kv = (KeyValue<Object, Object>) sourceRecord.getValue();
+ key = kv.getKey();
+ value = kv.getValue();
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue();
+ }
+ final Schema keySchema = getKafkaConnectSchemaForObject(key, defaultKeySchema);
+ final Schema valueSchema = getKafkaConnectSchemaForObject(value, defaultValueSchema);
+
+ long offset = taskContext.currentOffset(topicName, partition).incrementAndGet();
Review comment:
@sijie The Record<> may or may not contain sequenceID (Optional)
sequenceID, at the best, is `Optional.of(FunctionCommon.getSequenceId(message.getMessageId()))` - as i understand this applies to persistent topics only.
Record may or may not contain Message (Optional): sourceRecord.getMessage() which may return sequenceID or -1 (kafka has a thing against negative offsets, SinkRecord does not allow them).
MessageId can return byte[] (sourceRecord.getMessage().get().getMessageId().toByteArray()) of something - not guaranteed to be offset, rather a byte[] of anything, i.e. ledgerId, entryId, batchSize, topicName etc. Or it can throw UnsupportedOperationException
AFAICT usable sequenceID can be retrieved from data produced by kafka source (and there is absolutely no point in connecting kafka source to kafka sink in pulsar) or from the data read from bookkeeper.
I can do something along the lines of
```
long offset = sourceRecord.getRecordSequence()
.orElse(taskContext.currentOffset(topicName, partition).incrementAndGet());
```
This assumes that topic either has sequenceID for all records, or does not have them for all records.
Next problem is that kafka's offsets increase monotonically (per partition).
In case of BK, offsets is `(ledgerId << 28) | entryId` and is not guaranteed to be sequential, as I understand.
I don't know how kafka sinks react to that. Some sinks ignore offsets, others may track specific invariants.
Simply assigning the offsets the way I do now is the simplest solution.
It won't allow Exactly-once processing if the kafka sink deduping/overwriting data by offset id (and not the key, and if it supports this) but it will work as AtLeast-once.
I.e. if everything restarts it won't re-assign the same offsets (so deduping sink won't overwrite older data) but in case of replay/retry it will assign new offsets to the data potentially processed by the sink.
What do you hink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r597338047
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
Review comment:
see https://github.com/apache/pulsar/pull/9927#discussion_r597269231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r595571915
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
Review comment:
I am not sure we want to return -1 here.
If we see a temporary error we should fail and let the system recover
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
Review comment:
IMHO we should have some explicit mapping of supported types.
Using reflection may lead to unexpected results in the future
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617674145
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
then we have a problem, because we would have to configure at least authentication a security related stuff.
In 2.8.0 SinkContext allows you to access PulsarAdmin, probably we should have something like that for PulsarClient.
we can address this problem in a separate patch from my point of view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r627858739
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
Review comment:
Draft of the change: https://github.com/apache/pulsar/pull/10498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617106327
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
@eolivelli PulsarOffsetBackingStore needs it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r626098728
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj - Object to get schema of.
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null && pulsarSchema.getSchemaInfo() != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null || pulsarSchema.getSchemaInfo() == null
+ ? "null"
+ : pulsarSchema.getSchemaInfo().toString())
+ + " object class "
+ + (obj == null ? "null" : obj.getClass().getCanonicalName()));
+ }
+ return result;
+ }
+
+
+ @SuppressWarnings("rawtypes")
+ private SinkRecord toSinkRecord(Record<GenericObject> sourceRecord) {
+ final int partition = sourceRecord.getPartitionIndex().orElse(0);
+ final String topic = sourceRecord.getTopicName().orElse(topicName);
+ final Object key;
+ final Object value;
+ final Schema keySchema;
+ final Schema valueSchema;
+
+ // sourceRecord is never instanceof KVRecord
+ // https://github.com/apache/pulsar/pull/10113
+ if (unwrapKeyValueIfAvailable && sourceRecord.getSchema() != null
+ && sourceRecord.getSchema().getSchemaInfo() != null
+ && sourceRecord.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ KeyValueSchema kvSchema = (KeyValueSchema) sourceRecord.getSchema();
+ KeyValue kv = (KeyValue) sourceRecord.getValue().getNativeObject();
+ key = kv.getKey();
+ value = kv.getValue();
+ keySchema = getKafkaConnectSchema(kvSchema.getKeySchema(), key);
+ valueSchema = getKafkaConnectSchema(kvSchema.getValueSchema(), value);
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue().getNativeObject();
+ keySchema = Schema.STRING_SCHEMA;
Review comment:
this code builds Kafka's SinkRecord.
Kafka's SinkRecord expect STRING_SCHEMA with String object (i.e. https://github.com/confluentinc/kafka-connect-hdfs/blob/d140586c87275283a5265913b4130a3fa16fd81f/src/test/java/io/confluent/connect/hdfs/TestWithMiniDFSCluster.java#L184)
I will handle `msg.hasBase64EncodedKey()` (bytes key) separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r626099988
##########
File path: pulsar-io/kafka-connect-adaptor-nar/pom.xml
##########
@@ -30,13 +30,40 @@
<artifactId>pulsar-io-kafka-connect-adaptor-nar</artifactId>
<name>Pulsar IO :: Kafka Connect Adaptor NAR</name>
- <dependencies>
- <dependency>
- <groupId>${project.groupId}</groupId>
- <artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
- <version>${project.version}</version>
- </dependency>
- </dependencies>
+ <profiles>
+ <profile>
+ <id>main</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
Review comment:
I wanted to avoid building extra nar (slower build) which is not really necessary in most of the cases.
I'll remove this and will add this into the documentation for the Sink later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r599158553
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
Review comment:
I don't think it makes a lot of sense currently. Looking at https://github.com/apache/kafka/blob/8d6ae33ac51e8878e441e83256da313dc9fb4227/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L317 the timeout is used to control poll for new messages, that would mean making pulsar support such functionality (i.e. via passed SinkContext).
This can be implemented as a separate PR/later or can be left out to motivate people to implement/use native pulsar sinks for the cases where such functionality is critical.
Meanwhile there is a sink-level timeout setting AFAIU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
lhotari merged pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r595714308
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
Review comment:
you need to specify the partition. You can get partition id from `Record.getPartitionId`.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
Review comment:
You might also need to another branch to check if `Record` is a `KVRecord`.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
+ KeyValue<Object, Object> kv = (KeyValue<Object, Object>) sourceRecord.getValue();
+ key = kv.getKey();
+ value = kv.getValue();
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue();
+ }
+ final Schema keySchema = getKafkaConnectSchemaForObject(key, defaultKeySchema);
+ final Schema valueSchema = getKafkaConnectSchemaForObject(value, defaultValueSchema);
+
+ long offset = taskContext.currentOffset(topicName, partition).incrementAndGet();
Review comment:
I am not sure if this is the correct logic. I believe this is the offset from the source partition. Hence, you should get the underlying Pulsar message, retrieve message-id and convert it to a Long number. You shouldn't assign an offset for it, which would result in an indeterministic behavior.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
Review comment:
final
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
Review comment:
A good practice is to pass the pulsar sink context to Kafka sink task context wrapper. It is a clear indicator that we have wrapped the pulsar sink context as a Kafka sink task context even we don't actually use it.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
Review comment:
I think the logic here problematic. because the pendingFlush is not accessed in a `synchronized` block.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
Review comment:
Based on Kafka sink mode, you should only run the Pulsar source with failover subscription mode. We need to validate that. Because running a Kafka sink in shared subscription doesn't match Kafka's model
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkContext.java
##########
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.connect.connector.ConnectorContext;
+
+@Slf4j
+public class PulsarKafkaSinkContext implements ConnectorContext {
+
+ @Override
+ public void requestTaskReconfiguration() {
+ // noop;
Review comment:
throw unsupported exception to make the failure more explicit
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
Review comment:
same comment above.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
+ KeyValue<Object, Object> kv = (KeyValue<Object, Object>) sourceRecord.getValue();
+ key = kv.getKey();
+ value = kv.getValue();
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue();
+ }
+ final Schema keySchema = getKafkaConnectSchemaForObject(key, defaultKeySchema);
+ final Schema valueSchema = getKafkaConnectSchemaForObject(value, defaultValueSchema);
+
+ long offset = taskContext.currentOffset(topicName, partition).incrementAndGet();
+ SinkRecord sinkRecord = new SinkRecord(topicName,
+ partition,
+ keySchema,
+ key,
+ valueSchema,
+ value,
+ offset,
+ sourceRecord.getEventTime().orElse(null),
Review comment:
if event time is not specified, we should consider using publish time.
Or having a configuration to allow people configure which timestamp to use.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
Review comment:
I am not sure we need the default key and value schema here. A pulsar record carries the pulsar schema information. We should just write a util function to convert pulsar schema to kafka schema. If there is no schema, then use Kafka bytes schema.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private int batchSize = 1;
Review comment:
why batchSize is 1?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
Review comment:
I also don't think you need to register the callback here. Instead, I think you need to track the pulsar source records in the context in order to make sure the "offsets" are correctly recorded. once you call kafka sink to flush, you need to call sink context to flush. If you successfully flush, you can call sourceRecord to ack, otherwise you fail the source records.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
Review comment:
This should be implemented. If the offset (messageId) is reset, you need to reset the cursor of the input topic.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
Review comment:
The offset management here is completely wrong. It would result in a lot of duplicates and inconsistent behavior when sink connectors failed. You shouldn't be responsible for assigning offsets. The offsets are coming from the Pulsar records. Here you should manage the “offsets" (messageIds) of the records a Pulsar sink receives from the pulsar input topic.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
+ final Object key;
+ final Object value;
+ if (unwrapKeyValueIfAvailable && sourceRecord.getValue() instanceof KeyValue) {
+ KeyValue<Object, Object> kv = (KeyValue<Object, Object>) sourceRecord.getValue();
+ key = kv.getKey();
+ value = kv.getValue();
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue();
+ }
+ final Schema keySchema = getKafkaConnectSchemaForObject(key, defaultKeySchema);
+ final Schema valueSchema = getKafkaConnectSchemaForObject(value, defaultValueSchema);
+
+ long offset = taskContext.currentOffset(topicName, partition).incrementAndGet();
+ SinkRecord sinkRecord = new SinkRecord(topicName,
+ partition,
+ keySchema,
+ key,
+ valueSchema,
+ value,
+ offset,
+ sourceRecord.getEventTime().orElse(null),
+ TimestampType.NO_TIMESTAMP_TYPE);
Review comment:
The timestamp type should be determined base on what timestamp to use.
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
Review comment:
you should implement this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r597335899
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private int batchSize = 1;
Review comment:
In kafka producer the batch size is 16384 *bytes* by default.
Here it is number of messages the sink processes before flush. It is safer to set it to 1 and let user adjust it based on the VM size/type of kafka connect sink etc to avoid OOM or long delays between flushes.
I don't serialize the data before passing it to the kafka connect sink, and .getMessage() (that can get byte[]) is 1. optional and 2. potentially copies data internally.
I'll adjust the help message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-800539993
@yuvalgut thank you!
I postponed dealing with GenericRecords for now to make use of the changes @eolivelli is working on, i.e. https://github.com/apache/pulsar/issues/9844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r626117449
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ SinkContext ctx,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e);
+ } catch (ExecutionException e) {
+ log.error("error getting initial state of {}", topicPartition, e);
+ throw new RuntimeException("error getting initial state of " + topicPartition, e); }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
+ log.warn("timeout() is called but is not supported currently.");
+ }
+
+ @Override
+ public Set<TopicPartition> assignment() {
+ return currentOffsets.keySet();
+ }
+
+ @Override
+ public void pause(TopicPartition... topicPartitions) {
+ log.warn("pause() is called but is not supported currently.");
+ }
+
+ @Override
+ public void resume(TopicPartition... topicPartitions) {
+ log.warn("resume() is called but is not supported currently.");
+ }
+
+ @Override
+ public void requestCommit() {
+ log.warn("requestCommit() is called but is not supported currently.");
+ }
+
+ public void flushOffsets(Map<TopicPartition, OffsetAndMetadata> offsets) throws Exception {
+ Map<ByteBuffer, ByteBuffer> offsetMap = Maps.newHashMapWithExpectedSize(offsets.size());
+
+ offsets.forEach((tp, om) -> fillOffsetMap(offsetMap, tp, om.offset()));
+ CompletableFuture<Void> result = new CompletableFuture();
+ offsetStore.set(offsetMap, (ex, ignore) -> {
+ if (ex == null) {
+ result.complete(null);
+ } else {
+ log.error("error flushing offsets for {}",
+ Joiner.on(",").withKeyValueSeparator("=").join(offsets), ex);
+ result.completeExceptionally(ex);
+ }
+ });
+ result.get();
Review comment:
Follows the blocking style of SinkTask.flush().
Along with the SinkTask.flush() it runs on the same thread (flush is async for the connector) and 1. I don't want it to potentially concurrently with another flush 2. it is either get() here or there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r616834583
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,321 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.KVRecord;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
Review comment:
can we validate that `configs` size is 1 ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,321 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.KVRecord;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null ? "null" : pulsarSchema.getSchemaInfo().toString())
Review comment:
getSchemaInfo() may return null
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ SinkContext ctx,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
+ // noop
+ }
+
+ @Override
+ public Set<TopicPartition> assignment() {
+ return currentOffsets.keySet();
+ }
+
+ @Override
+ public void pause(TopicPartition... topicPartitions) {
+ // noop
+ }
+
+ @Override
+ public void resume(TopicPartition... topicPartitions) {
+ // noop
+ }
+
+ @Override
+ public void requestCommit() {
+ // noop
+ }
+
+ public void flushOffsets(Map<TopicPartition, OffsetAndMetadata> offsets) throws Exception {
+ Map<ByteBuffer, ByteBuffer> offsetMap = Maps.newHashMapWithExpectedSize(offsets.size());
+
+ offsets.forEach((tp, om) -> fillOffsetMap(offsetMap, tp, om.offset()));
+ CompletableFuture<Void> result = new CompletableFuture();
+ offsetStore.set(offsetMap, (ex, ignore) -> {
+ if (ex == null) {
+ result.complete(null);
+ } else {
+ log.error("error flushing offsets", ex);
Review comment:
can we log more information, like the offsets we wanted to commit ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/SchemaedFileStreamSinkTask.java
##########
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.file.FileStreamSinkTask;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.testng.collections.Maps;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public class SchemaedFileStreamSinkTask extends FileStreamSinkTask {
+
+ @Override
+ public void put(Collection<SinkRecord> sinkRecords) {
+
+ List<SinkRecord> out = Lists.newLinkedList();
+
+ Iterator var2 = sinkRecords.iterator();
+
+ while (var2.hasNext()) {
+ SinkRecord record = (SinkRecord) var2.next();
+
+ Object val = record.valueSchema() == Schema.BYTES_SCHEMA
+ ? new String((byte[])record.value(), StandardCharsets.US_ASCII)
+ : record.value();
+
+ Map<String, Object> recOut = Maps.newHashMap();
+ recOut.put("keySchema", record.keySchema().type().toString());
+ recOut.put("valueSchema", record.valueSchema().type().toString());
+ recOut.put("key", record.key());
+ recOut.put("value", val);
+
+ ObjectMapper om = new ObjectMapper();
+ try {
+ SinkRecord toSink = new SinkRecord(record.topic(),
Review comment:
why are you cloning the SinkRecord ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/SchemaedFileStreamSinkTask.java
##########
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.file.FileStreamSinkTask;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.testng.collections.Maps;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public class SchemaedFileStreamSinkTask extends FileStreamSinkTask {
Review comment:
can we write a line of JavaDoc in order to explain what it is this class ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
Review comment:
@sijie can we do this in a follow up work ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkContext.java
##########
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.connect.connector.ConnectorContext;
+
+@Slf4j
+public class PulsarKafkaSinkContext implements ConnectorContext {
+
+ @Override
+ public void requestTaskReconfiguration() {
+ throw new UnsupportedOperationException("not implemented");
+ }
+
+ @Override
+ public void raiseError(Exception e) {
+ throw new UnsupportedOperationException("not implemented");
Review comment:
can we at least log the exception (with stacktrace) ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
why do we need this value ? we are running inside Pulsar
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
Review comment:
can we make this default an empty string and make it required (fail if it is not configured) ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
Review comment:
can we make this default an empty string and make it required (fail if it is not configured) ?
or if we do not make this required, can we leave it blank and compute a name based on the `topic` configuration parameter ?
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/SchemaedFileStreamSinkTask.java
##########
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.file.FileStreamSinkTask;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.testng.collections.Maps;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public class SchemaedFileStreamSinkTask extends FileStreamSinkTask {
+
+ @Override
+ public void put(Collection<SinkRecord> sinkRecords) {
+
+ List<SinkRecord> out = Lists.newLinkedList();
+
+ Iterator var2 = sinkRecords.iterator();
Review comment:
<SinkRecord>
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/GenericObjectWrapper.java
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.kafka.connect;
+
+import lombok.Builder;
+import lombok.Data;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.List;
+
+/**
+ * org.apache.pulsar.client.impl.schema.GenericObjectWrapper is not public
+ * (and is not intended to be, at least for now).
+ * this one implements similar functionality, for testing.
+ */
+@Data
+@Builder
+class GenericObjectWrapper implements GenericRecord {
Review comment:
what about naming this MockGenericObjectWrapper ? (otherwise in the IDE everytime someone looks for GenericObjectWrapper he will find this class as well)
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
+ map.forEach((key, value) -> {
+ if (!currentOffsets.containsKey(key)) {
+ runRepartition.set(true);
+ }
+ currentOffsets.put(key, new AtomicLong(value));
+ });
+
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ }
+
+ @Override
+ public void offset(TopicPartition topicPartition, long l) {
+ Map<TopicPartition, Long> map = Maps.newHashMap();
+ map.put(topicPartition, l);
+ this.offset(map);
+ }
+
+ @Override
+ public void timeout(long l) {
Review comment:
I would create an issue for further work
we could log a line at "error" level, this way if a Connector uses this feature it will be logged and the user will probably see it
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
Review comment:
makes sense to me, although probably it is not needed, at least for this patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yuvalgut commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
yuvalgut commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-800536901
cool! that will be really useful.
for supporting avro we can check if the Object is type of org.apache.pulsar.client.impl.schema.generic.GenericAvroRecord and then get hold of the actual avro record with '.getAvroRecord()'.
io.confluent.connect.avro.AvroData will let us to convert the avro into kafka SchemaAndValue:
avroData.toConnectData(avroRecord.getSchema(), avroRecord) and finally generate kafka SinkRecord.
just throwing some ideas...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617829977
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
Sounds good to me, but this is to be done in a separate patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-827800494
@eolivelli I don't think it is ready for merge. Please see comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617724217
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
> In 2.8.0 SinkContext allows you to access PulsarAdmin
@eolivelli I looked at this and it is somewhat correct. Somewhat is because:
1. PulsarAdmin is exposed only if config.isExposePulsarAdminClientEnabled() is set.
2. One has to cast received instance of SinkContext to Context, assuming that SinkContext implementation also implements Context, because what we have is:
```
interface SinkContext extends ConnectorContext
interface SourceContext extends ConnectorContext
interface ConnectorContext (in org.apache.pulsar.io.core, doesn't extend anything)
interface Context (in org.apache.pulsar.functions.api, doesn't extend anything, the one with getPulsarAdmin())
class ContextImpl implements Context, SinkContext, SourceContext, AutoCloseable
```
So any access to getPulsarAdmin() and other API from Context interface in sink/source would need to do instanceof check & cast to Context which enforces assumption that any SinkContext/SourceContext also is a Context.
This results in a few problems: discoverability of the API (i.e. in IDE), cannot change i.e. SinkContext implementation and assume it works because there is implicit requirement for any SinkContext impl to also implement a few other interfaces etc.
Can we just make it explicit and move some or all of these methods into the ConnectorContext, make Context extend ConnectorContext?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r597323899
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
Review comment:
Maybe I misunderstood something but:
SubscriptionType is not passed as part of the context. Existing Pulsar Kafka Sink (KafkaBytesSink) does not enforce specific subscription model so I'll skip dealing with wiring this info through the context and enforcing it (in both sinks) for after this PR is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r596433829
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().entrySet()
+ .forEach(kv -> props.put(kv.getKey(), kv.getValue()));
+
+ defaultKeySchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultKeySchema()).get(null);
+ defaultValueSchema = (Schema)Schema.class
+ .getField(kafkaSinkConfig.getDefaultValueSchema()).get(null);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || numPendingRecords.get() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ numPendingRecords.get(), batchSize);
+ }
+
+ if (numPendingRecords.getAndSet(0) == 0) {
+ return;
+ }
+
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+ CompletableFuture<Void> flushCf;
+ synchronized (this) {
+ flushCf = pendingFlush;
+ pendingFlush = new CompletableFuture<>();
+ }
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ flushCf.complete(null);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ flushCf.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ public static Schema getKafkaConnectSchemaForObject(Object obj, Schema defaultSchema) {
+ if (obj == null) {
+ return defaultSchema;
+ }
+
+ if (primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+
+ // Other types are not supported yet.
+ // Will fallback to defaults provided.
+ return defaultSchema;
+ }
+
+ private SinkRecord toSinkRecord(Record<Object> sourceRecord) {
+ final int partition = 0;
Review comment:
Record : `default Optional<String> getPartitionId()` - this will not work.
I created https://github.com/apache/pulsar/pull/9947 - let me know if you ok with this.
in this case I can do something like `final int partition = sourceRecord.getPartitionNumber().orElse(0);`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-843567096
@sijie please take another look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r624253160
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<GenericObject> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ private final static ImmutableMap<SchemaType, Schema> pulsarSchemaTypeTypeToKafkaSchema;
+
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ pulsarSchemaTypeTypeToKafkaSchema = ImmutableMap.<SchemaType, Schema>builder()
+ .put(SchemaType.BOOLEAN, Schema.BOOLEAN_SCHEMA)
+ .put(SchemaType.INT8, Schema.INT8_SCHEMA)
+ .put(SchemaType.INT16, Schema.INT16_SCHEMA)
+ .put(SchemaType.INT32, Schema.INT32_SCHEMA)
+ .put(SchemaType.INT64, Schema.INT64_SCHEMA)
+ .put(SchemaType.FLOAT, Schema.FLOAT32_SCHEMA)
+ .put(SchemaType.DOUBLE, Schema.FLOAT64_SCHEMA)
+ .put(SchemaType.STRING, Schema.STRING_SCHEMA)
+ .put(SchemaType.BYTES, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final ConcurrentLinkedDeque<Record<GenericObject>> pendingFlushQueue = new ConcurrentLinkedDeque<>();
+ private volatile boolean isRunning = false;
+
+ private final Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<GenericObject> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlushQueue.add(sourceRecord);
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
+ kafkaSinkConfig = PulsarKafkaConnectSinkConfig.load(config);
+ Objects.requireNonNull(kafkaSinkConfig.getTopic(), "Kafka topic is not set");
+ topicName = kafkaSinkConfig.getTopic();
+ unwrapKeyValueIfAvailable = kafkaSinkConfig.isUnwrapKeyValueIfAvailable();
+
+ String kafkaConnectorFQClassName = kafkaSinkConfig.getKafkaConnectorSinkClass();
+ kafkaSinkConfig.getKafkaConnectorConfigProperties().forEach(props::put);
+
+ Class<?> clazz = Class.forName(kafkaConnectorFQClassName);
+ connector = (SinkConnector) clazz.getConstructor().newInstance();
+
+ Class<? extends Task> taskClass = connector.taskClass();
+ sinkContext = new PulsarKafkaSinkContext();
+ connector.initialize(sinkContext);
+ connector.start(Maps.fromProperties(props));
+
+ List<Map<String, String>> configs = connector.taskConfigs(1);
+ Preconditions.checkNotNull(configs);
+ Preconditions.checkArgument(configs.size() == 1);
+
+ configs.forEach(x -> {
+ x.put(OFFSET_STORAGE_TOPIC_CONFIG, kafkaSinkConfig.getOffsetStorageTopic());
+ x.put(PULSAR_SERVICE_URL_CONFIG, kafkaSinkConfig.getPulsarServiceUrl());
+ });
+ task = (SinkTask) taskClass.getConstructor().newInstance();
+ taskContext =
+ new PulsarKafkaSinkTaskContext(configs.get(0), ctx, task::open);
+ task.initialize(taskContext);
+ task.start(configs.get(0));
+
+ batchSize = kafkaSinkConfig.getBatchSize();
+ lingerMs = kafkaSinkConfig.getLingerTimeMs();
+ scheduledExecutor.scheduleAtFixedRate(() ->
+ this.flushIfNeeded(true), lingerMs, lingerMs, TimeUnit.MILLISECONDS);
+
+
+ isRunning = true;
+ log.info("Kafka sink started : {}.", props);
+ }
+
+ private void flushIfNeeded(boolean force) {
+ if (force || pendingFlushQueue.stream().limit(batchSize).count() >= batchSize) {
+ scheduledExecutor.submit(this::flush);
+ }
+ }
+
+ // flush always happens on the same thread
+ public void flush() {
+ if (log.isDebugEnabled()) {
+ log.debug("flush requested, pending: {}, batchSize: {}",
+ pendingFlushQueue.size(), batchSize);
+ }
+
+ if (pendingFlushQueue.isEmpty()) {
+ return;
+ }
+
+ final Record<GenericObject> lastNotFlushed = pendingFlushQueue.getLast();
+ Map<TopicPartition, OffsetAndMetadata> currentOffsets = taskContext.currentOffsets();
+
+ try {
+ task.flush(currentOffsets);
+ taskContext.flushOffsets(currentOffsets);
+ ackUntil(lastNotFlushed, Record::ack);
+ } catch (Throwable t) {
+ log.error("error flushing pending records", t);
+ ackUntil(lastNotFlushed, Record::fail);
+ }
+ }
+
+ private void ackUntil(Record<GenericObject> lastNotFlushed, java.util.function.Consumer<Record<GenericObject>> cb) {
+ while (!pendingFlushQueue.isEmpty()) {
+ Record<GenericObject> r = pendingFlushQueue.pollFirst();
+ cb.accept(r);
+ if (r == lastNotFlushed) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * org.apache.kafka.connect.data.Schema for the object
+ * @param obj - Object to get schema of.
+ * @return org.apache.kafka.connect.data.Schema
+ */
+ private static Schema getKafkaConnectSchemaForObject(Object obj) {
+ if (obj != null && primitiveTypeToSchema.containsKey(obj.getClass())) {
+ return primitiveTypeToSchema.get(obj.getClass());
+ }
+ return null;
+ }
+
+ public static Schema getKafkaConnectSchema(org.apache.pulsar.client.api.Schema pulsarSchema, Object obj) {
+ if (pulsarSchema != null && pulsarSchema.getSchemaInfo() != null
+ && pulsarSchemaTypeTypeToKafkaSchema.containsKey(pulsarSchema.getSchemaInfo().getType())) {
+ return pulsarSchemaTypeTypeToKafkaSchema.get(pulsarSchema.getSchemaInfo().getType());
+ }
+
+ Schema result = getKafkaConnectSchemaForObject(obj);
+ if (result == null) {
+ throw new IllegalStateException("Unsupported kafka schema for Pulsar Schema "
+ + (pulsarSchema == null || pulsarSchema.getSchemaInfo() == null
+ ? "null"
+ : pulsarSchema.getSchemaInfo().toString())
+ + " object class "
+ + (obj == null ? "null" : obj.getClass().getCanonicalName()));
+ }
+ return result;
+ }
+
+
+ @SuppressWarnings("rawtypes")
+ private SinkRecord toSinkRecord(Record<GenericObject> sourceRecord) {
+ final int partition = sourceRecord.getPartitionIndex().orElse(0);
+ final String topic = sourceRecord.getTopicName().orElse(topicName);
+ final Object key;
+ final Object value;
+ final Schema keySchema;
+ final Schema valueSchema;
+
+ // sourceRecord is never instanceof KVRecord
+ // https://github.com/apache/pulsar/pull/10113
+ if (unwrapKeyValueIfAvailable && sourceRecord.getSchema() != null
+ && sourceRecord.getSchema().getSchemaInfo() != null
+ && sourceRecord.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ KeyValueSchema kvSchema = (KeyValueSchema) sourceRecord.getSchema();
+ KeyValue kv = (KeyValue) sourceRecord.getValue().getNativeObject();
+ key = kv.getKey();
+ value = kv.getValue();
+ keySchema = getKafkaConnectSchema(kvSchema.getKeySchema(), key);
+ valueSchema = getKafkaConnectSchema(kvSchema.getValueSchema(), value);
+ } else {
+ key = sourceRecord.getKey().orElse(null);
+ value = sourceRecord.getValue().getNativeObject();
+ keySchema = Schema.STRING_SCHEMA;
+ valueSchema = getKafkaConnectSchema(sourceRecord.getSchema(), value);
+ }
+
+ long offset = sourceRecord.getRecordSequence()
+ .orElse(-1L);
+ if (offset < 0) {
+ offset = taskContext.currentOffset(topic, partition)
+ .incrementAndGet();
+ } else {
+ final long curr = offset;
+ taskContext.currentOffset(topic, partition)
+ .updateAndGet(curMax -> Math.max(curr, curMax));
+ }
+
+ Long timestamp = null;
+ TimestampType timestampType = TimestampType.NO_TIMESTAMP_TYPE;
+ if (sourceRecord.getEventTime().isPresent()) {
+ timestamp = sourceRecord.getEventTime().get();
+ timestampType = TimestampType.CREATE_TIME;
+ } else if (sourceRecord.getMessage().isPresent()) {
+ timestamp = sourceRecord.getMessage().get().getPublishTime();
+ timestampType = TimestampType.LOG_APPEND_TIME;
Review comment:
@sijie maybe it is better to leave it as TimestampType.NO_TIMESTAMP_TYPE in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617663837
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/SchemaedFileStreamSinkTask.java
##########
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.file.FileStreamSinkTask;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.testng.collections.Maps;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public class SchemaedFileStreamSinkTask extends FileStreamSinkTask {
+
+ @Override
+ public void put(Collection<SinkRecord> sinkRecords) {
+
+ List<SinkRecord> out = Lists.newLinkedList();
+
+ Iterator var2 = sinkRecords.iterator();
+
+ while (var2.hasNext()) {
+ SinkRecord record = (SinkRecord) var2.next();
+
+ Object val = record.valueSchema() == Schema.BYTES_SCHEMA
+ ? new String((byte[])record.value(), StandardCharsets.US_ASCII)
+ : record.value();
+
+ Map<String, Object> recOut = Maps.newHashMap();
+ recOut.put("keySchema", record.keySchema().type().toString());
+ recOut.put("valueSchema", record.valueSchema().type().toString());
+ recOut.put("key", record.key());
+ recOut.put("value", val);
+
+ ObjectMapper om = new ObjectMapper();
+ try {
+ SinkRecord toSink = new SinkRecord(record.topic(),
Review comment:
okay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617724217
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "Number of messages the sink processes before flush.")
+ private int batchSize = 1;
+
+ @FieldDoc(
+ defaultValue = "2147483647L",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private long lingerTimeMs = 2147483647L;
+
+ @FieldDoc(
+ defaultValue = "pulsar-io-adaptor-topic",
+ help = "The Kafka topic name that passed to kafka sink.")
+ private String topic = "pulsar-io-adaptor-topic";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "A kafka-connector sink class to use.")
+ private String kafkaConnectorSinkClass;
+
+ @FieldDoc(
+ defaultValue = "",
+ help = "Config properties to pass to the kafka connector.")
+ private Map<String, String> kafkaConnectorConfigProperties;
+
+ @FieldDoc(
+ defaultValue = "kafka-adaptor-sink-offsets",
+ help = "Pulsar topic to store offsets at.")
+ private String offsetStorageTopic = "kafka-adaptor-sink-offsets";
+
+ @FieldDoc(
+ required = true,
+ defaultValue = "",
+ help = "Pulsar service URL.")
+ private String pulsarServiceUrl;
Review comment:
> In 2.8.0 SinkContext allows you to access PulsarAdmin
@eolivelli I looked at this and it is somewhat correct. Somewhat is because:
1. PulsarAdmin is exposed only if config.isExposePulsarAdminClientEnabled() is set.
2. One has to cast received instance of SinkContext to Context, assuming that SinkContext implementation also implements Context, because what we have is:
```
interface SinkContext extends ConnectorContext
interface SourceContext extends ConnectorContext
interface ConnectorContext (in org.apache.pulsar.io.core, doesn't extend anything)
interface Context (in org.apache.pulsar.functions.api, doesn't extend anything, the one with getPulsarAdmin())
class ContextImpl implements Context, SinkContext, SourceContext, AutoCloseable
```
So any access to getPulsarAdmin() and other API from Context interface in sink/source would need to do instanceof check & cast to Context which enforces assumption that any SinkContext/SourceContext also is a Context.
This results in a few problems: discoverability of the API (i.e. in IDE), cannot change i.e. SinkContext implementation and assume it works because there is implicit requirement for any SinkContext impl to also implement a few other interfaces etc.
Can we just make it explicit and move some or all of these methods into the ConnectorContext, make Context extend ConnectorContext? (as a separate change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r617124058
##########
File path: pulsar-io/kafka-connect-adaptor/src/test/java/org/apache/pulsar/io/kafka/connect/SchemaedFileStreamSinkTask.java
##########
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.file.FileStreamSinkTask;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.testng.collections.Maps;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public class SchemaedFileStreamSinkTask extends FileStreamSinkTask {
+
+ @Override
+ public void put(Collection<SinkRecord> sinkRecords) {
+
+ List<SinkRecord> out = Lists.newLinkedList();
+
+ Iterator var2 = sinkRecords.iterator();
+
+ while (var2.hasNext()) {
+ SinkRecord record = (SinkRecord) var2.next();
+
+ Object val = record.valueSchema() == Schema.BYTES_SCHEMA
+ ? new String((byte[])record.value(), StandardCharsets.US_ASCII)
+ : record.value();
+
+ Map<String, Object> recOut = Maps.newHashMap();
+ recOut.put("keySchema", record.keySchema().type().toString());
+ recOut.put("valueSchema", record.valueSchema().type().toString());
+ recOut.put("key", record.key());
+ recOut.put("value", val);
+
+ ObjectMapper om = new ObjectMapper();
+ try {
+ SinkRecord toSink = new SinkRecord(record.topic(),
Review comment:
@eolivelli I am changing value (`om.writeValueAsString(recOut)` below) and original SinkRecord is immutable. This is for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r623503962
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSink.java
##########
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.sink.SinkConnector;
+import org.apache.kafka.connect.sink.SinkRecord;
+import org.apache.kafka.connect.sink.SinkTask;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.KeyValue;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG;
+
+@Slf4j
+public class KafkaConnectSink implements Sink<Object> {
+
+ private boolean unwrapKeyValueIfAvailable;
+
+ private final static ImmutableMap<Class<?>, Schema> primitiveTypeToSchema;
+ static {
+ primitiveTypeToSchema = ImmutableMap.<Class<?>, Schema>builder()
+ .put(Boolean.class, Schema.BOOLEAN_SCHEMA)
+ .put(Byte.class, Schema.INT8_SCHEMA)
+ .put(Short.class, Schema.INT16_SCHEMA)
+ .put(Integer.class, Schema.INT32_SCHEMA)
+ .put(Long.class, Schema.INT64_SCHEMA)
+ .put(Float.class, Schema.FLOAT32_SCHEMA)
+ .put(Double.class, Schema.FLOAT64_SCHEMA)
+ .put(String.class, Schema.STRING_SCHEMA)
+ .put(byte[].class, Schema.BYTES_SCHEMA)
+ .build();
+ }
+
+ private PulsarKafkaSinkContext sinkContext;
+ private PulsarKafkaSinkTaskContext taskContext;
+ private SinkConnector connector;
+ private SinkTask task;
+
+ private Schema defaultKeySchema;
+ private Schema defaultValueSchema;
+
+
+ private int batchSize;
+ private long lingerMs;
+ private final ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder()
+ .setNameFormat("pulsar-io-kafka-adaptor-sink-flush-%d")
+ .build());
+ private final AtomicInteger numPendingRecords = new AtomicInteger(0);
+
+ private volatile CompletableFuture<Void> pendingFlush = new CompletableFuture<>();
+ private volatile boolean isRunning = false;
+
+ private Properties props = new Properties();
+ private PulsarKafkaConnectSinkConfig kafkaSinkConfig;
+
+ protected String topicName;
+
+ @Override
+ public void write(Record<Object> sourceRecord) {
+ if (log.isDebugEnabled()) {
+ log.debug("Record sending to kafka, record={}.", sourceRecord);
+ }
+
+ if (!isRunning) {
+ log.error("Sink is stopped. Cannot send the record {}", sourceRecord);
+ sourceRecord.fail();
+ return;
+ }
+
+ try {
+ SinkRecord record = toSinkRecord(sourceRecord);
+ task.put(Lists.newArrayList(record));
+ } catch (Exception ex) {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ return;
+ }
+ pendingFlush.whenComplete((ignore, ex) -> {
+ if (ex == null) {
+ sourceRecord.ack();
+ } else {
+ log.error("Error sending the record {}", sourceRecord, ex);
+ sourceRecord.fail();
+ }
+ throw new IllegalArgumentException();
+ });
+ numPendingRecords.incrementAndGet();
+ flushIfNeeded(false);
+ }
+
+ @Override
+ public void close() throws Exception {
+ isRunning = false;
+ flushIfNeeded(true);
+ scheduledExecutor.shutdown();
+ if (!scheduledExecutor.awaitTermination(10 * lingerMs, TimeUnit.MILLISECONDS)) {
+ log.error("scheduledExecutor did not terminate in {} ms", 10 * lingerMs);
+ }
+
+ task.stop();
+ connector.stop();
+ taskContext.close();
+
+ log.info("Kafka sink stopped.");
+ }
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext ctx) throws Exception {
Review comment:
This one will need https://github.com/apache/pulsar/pull/10446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r634751192
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
Review comment:
@sijie The PR is merged; I rebased on top of the PR and implemented pause/resume/offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r626187058
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaSinkTaskContext.java
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.connect.sink.SinkTaskContext;
+import org.apache.kafka.connect.storage.OffsetBackingStore;
+
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.pulsar.io.kafka.connect.PulsarKafkaWorkerConfig.TOPIC_NAMESPACE_CONFIG;
+
+@Slf4j
+public class PulsarKafkaSinkTaskContext implements SinkTaskContext {
+
+ private final Map<String, String> config;
+
+ private final OffsetBackingStore offsetStore;
+ private final String topicNamespace;
+ private final Consumer<Collection<TopicPartition>> onPartitionChange;
+ private final AtomicBoolean runRepartition = new AtomicBoolean(false);
+
+ private final ConcurrentHashMap<TopicPartition, AtomicLong> currentOffsets = new ConcurrentHashMap<>();
+
+ public PulsarKafkaSinkTaskContext(Map<String, String> config,
+ Consumer<Collection<TopicPartition>> onPartitionChange) {
+ this.config = config;
+
+ offsetStore = new PulsarOffsetBackingStore();
+ PulsarKafkaWorkerConfig pulsarKafkaWorkerConfig = new PulsarKafkaWorkerConfig(config);
+ offsetStore.configure(pulsarKafkaWorkerConfig);
+ offsetStore.start();
+
+ this.onPartitionChange = onPartitionChange;
+ this.topicNamespace = pulsarKafkaWorkerConfig.getString(TOPIC_NAMESPACE_CONFIG);
+ }
+
+ public void close() {
+ offsetStore.stop();
+ }
+
+ @Override
+ public Map<String, String> configs() {
+ return config;
+ }
+
+ public AtomicLong currentOffset(String topic, int partition) {
+ return currentOffset(new TopicPartition(topic, partition));
+ }
+
+ public AtomicLong currentOffset(TopicPartition topicPartition) {
+ AtomicLong offset = currentOffsets.computeIfAbsent(topicPartition, kv -> {
+ List<ByteBuffer> req = Lists.newLinkedList();
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ req.add(key);
+ CompletableFuture<Long> offsetFuture = new CompletableFuture<>();
+ offsetStore.get(req, (Throwable ex, Map<ByteBuffer, ByteBuffer> result) -> {
+ if (ex == null) {
+ if (result != null && result.size() != 0) {
+ Optional<ByteBuffer> val = result.entrySet().stream()
+ .filter(entry -> entry.getKey().equals(key))
+ .findFirst().map(entry -> entry.getValue());
+ if (val.isPresent()) {
+ long received = val.get().getLong();
+ if (log.isDebugEnabled()) {
+ log.debug("read initial offset for {} == {}", topicPartition, received);
+ }
+ offsetFuture.complete(received);
+ return;
+ }
+ }
+ offsetFuture.complete(-1L);
+ } else {
+ offsetFuture.completeExceptionally(ex);
+ }
+ });
+
+ runRepartition.set(true);
+ try {
+ return new AtomicLong(offsetFuture.get());
+ } catch (Exception e) {
+ log.error("error getting initial state of " + topicPartition.toString(), e);
+ return new AtomicLong(-1L);
+ }
+ });
+ if (runRepartition.compareAndSet(true, false)) {
+ onPartitionChange.accept(currentOffsets.keySet());
+ }
+ return offset;
+ }
+
+ public Map<TopicPartition, OffsetAndMetadata> currentOffsets() {
+
+ Map<TopicPartition, OffsetAndMetadata> snapshot = Maps.newHashMapWithExpectedSize(currentOffsets.size());
+ currentOffsets.forEach((topicPartition, offset) -> {
+ if (offset.get() > 0) {
+ snapshot.put(topicPartition,
+ new OffsetAndMetadata(offset.get(), Optional.empty(), null));
+ }
+ });
+ return snapshot;
+ }
+
+ private ByteBuffer topicPartitionAsKey(TopicPartition topicPartition) {
+ return ByteBuffer.wrap((topicNamespace + "/" + topicPartition.toString()).getBytes(UTF_8));
+
+ }
+
+ private void fillOffsetMap(Map<ByteBuffer, ByteBuffer> offsetMap, TopicPartition topicPartition, long l) {
+ ByteBuffer key = topicPartitionAsKey(topicPartition);
+ ByteBuffer value = ByteBuffer.allocate(Long.BYTES);
+ value.putLong(l);
+ value.flip();
+ offsetMap.put(key, value);
+ }
+
+ @SneakyThrows
+ @Override
+ public void offset(Map<TopicPartition, Long> map) {
Review comment:
Looking at this closer, to actually reset the consumer (and `pause()`/`resume()`) properly we'll need a way for the sink to communicate to the source. I'll think about it/draft the implementation but I think it can be done as a follow up.
Not every kafka sink actually uses it AFAICT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#discussion_r624280548
##########
File path: pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.io.kafka.connect;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import lombok.Data;
+import lombok.experimental.Accessors;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+@Data
+@Accessors(chain = true)
+public class PulsarKafkaConnectSinkConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ @FieldDoc(
+ defaultValue = "1",
+ help = "The batch size that Kafka producer will attempt to batch records together.")
+ private int batchSize = 1;
Review comment:
@sijie this would involve doing something like `sourceRecord.getMessage().get().getData().length`
`MessageImpl.getData()` can end up allocating new array. :/
I'll add something like getDataSize() to the Message interface (separate PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #9927: Add ability to use Kafka's sinks as pulsar sinks
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #9927:
URL: https://github.com/apache/pulsar/pull/9927#issuecomment-800524060
this replaces https://github.com/apache/pulsar/pull/9825 - reworked according to @sijie 's feedback
@sijie and @eolivelli please take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org