You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2016/08/14 18:44:03 UTC

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

GitHub user rmetzger opened a pull request:

    https://github.com/apache/flink/pull/2369

    [FLINK-4035] Add a streaming connector for Apache Kafka 0.10.x

    This pull request subsumes https://github.com/apache/flink/pull/2231.
    
    Compared to #2231, I've based the connector on the existing 0.9 code (by extending it, reducing the amount of copied code), added a test case for the timestamp functionality and updated the documentation.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rmetzger/flink flink4035_rebased

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2369.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2369
    
----
commit e3b2ede004b3c4ab6e37df1d6a268a52c1565316
Author: radekg <ra...@gruchalski.com>
Date:   2016-07-12T17:19:01Z

    [FLINK-4035] Add support for Kafka 0.10.x.

commit 9d358a9ad5bcdd1ea644bc5b902240423586faf3
Author: Robert Metzger <rm...@apache.org>
Date:   2016-08-09T14:38:21Z

    [FLINK-4035] Refactor the Kafka 0.10 connector to be based upon the 0.9 connector
    
    Add a test case for Kafka's new timestamp functionality and update the documentation.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Left a few comments on some high-level design choices for a first review. Mostly on `FlinkKafkaProducer010`, I wonder if there are other better possibilities over there?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Looks like we need to rebase this PR on the recently merged Kerberos support.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r80496915
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka010Fetcher.java ---
    @@ -60,12 +60,17 @@ protected void assignPartitionsToConsumer(KafkaConsumer<byte[], byte[]> consumer
     		consumer.assign(topicPartitions);
     	}
     
    +	@Override
    +	protected void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partition, long offset, ConsumerRecord consumerRecord) throws Exception {
    +		// pass timestamp
    +		super.emitRecord(record, partition, offset, consumerRecord.timestamp());
    +	}
    +
     	/**
     	 * Emit record Kafka-timestamp aware.
     	 */
     	@Override
    -	protected <R> void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, R kafkaRecord) throws Exception {
    -		long timestamp = ((ConsumerRecord) kafkaRecord).timestamp();
    +	protected void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, long timestamp) throws Exception {
     		if (timestampWatermarkMode == NO_TIMESTAMPS_WATERMARKS) {
    --- End diff --
    
    Yes, I'll do that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r80491475
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,399 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.common.functions.IterationRuntimeContext;
    +import org.apache.flink.api.common.functions.RichFunction;
    +import org.apache.flink.api.common.functions.RuntimeContext;
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.functions.sink.SinkFunction;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.getPropertiesFromBrokerList;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This producer is a hybrid between a regular regular sink function (a)
    + * and a custom operator (b).
    + *
    + * For (a), the class implements the SinkFunction and RichFunction interfaces.
    + * For (b), it extends the StreamTask class.
    + *
    + * Details about approach (a):
    + *
    + *  Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the
    + *  DataStream.addSink() method.
    + *  Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record
    + *  the Kafka 0.10 producer has a section invocation option, approach (b).
    + *
    + * Details about approach (b):
    + *  Kafka 0.10 supports writing the timestamp attached to a record to Kafka. When adding the
    + *  FlinkKafkaProducer010 using the FlinkKafkaProducer010.writeToKafka() method, the Kafka producer
    + *  can access the internal record timestamp of the record and write it to Kafka.
    + *
    + * All methods and constructors in this class are marked with the approach they are needed for.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> implements SinkFunction<T>, RichFunction {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for timestamp writing ------------------
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										KeyedSerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    --- End diff --
    
    Its a good idea, I'll rename the methods


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/2369


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by cjstehno <gi...@git.apache.org>.
Github user cjstehno commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Any thoughts on when this might make it into a release? We are having issues running Flink with Kafka 0.10 and would like to have an idea of whether we can/should wait for this or pull it and try building our own. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75572263
  
    --- Diff: docs/apis/streaming/connectors/kafka.md ---
    @@ -291,3 +301,35 @@ higher value.
     There is currently no transactional producer for Kafka, so Flink can not guarantee exactly-once delivery
     into a Kafka topic.
     
    +### Using Kafka timestamps and Flink event time in Kafka 0.10
    +
    +Since Apache Kafka 0.10., Kafka's messages can carry [timestamps](https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message), indicating
    +the time the event has occurred (see ["event time" in Apache Flink](../event_time.html)) or the time when the message
    +has been written to the Kafka broker.
    +
    +The `FlinkKafkaConsumer010` will emit records with the timestamp attached, if the time characteristic in Flink is 
    +set to `TimeCharacteristic.EventTime` (`StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)`).
    +
    +The Kafka consumer does not emit watermarks. To emit watermarks, the same mechanisms as described above in 
    +"Kafka Consumers and Timestamp Extraction/Watermark Emission"  using the `assignTimestampsAndWatermarks` method are applicable.
    +
    +There is no need to define a timestamp extractor when using the timestamps from Kafka. The `previousElementTimestamp` argument of 
    +the `extractTimestamp()` method contains the timestamp carried by the Kafka message.
    +
    +A timestamp extractor for a Kafka consumer would look like this:
    +{% highlight java %}
    +public long extractTimestamp(Long element, long previousElementTimestamp) {
    +    return previousElementTimestamp;
    +}
    +{% endhighlight %}
    +
    +
    +
    +The `FlinkKafkaProducer010` only emits the record timestamp, if `setWriteTimestampToKafka(true)` is set.
    +
    +{% highlight java %}
    +FlinkKafkaProducer010.FlinkKafkaProducer010Configuration config = FlinkKafkaProducer010.writeToKafka(streamWithTimestamps, topic, new SimpleStringSchema(), standardProps);
    +config.setWriteTimestampToKafka(true);
    +{% endhighlight %}
    --- End diff --
    
    I find the usage pattern of this a bit unfamiliar. I've explained this in inline comments of the `FlinkKafkaProducer010` class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75903103
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka010Fetcher.java ---
    @@ -0,0 +1,91 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka.internal;
    +
    +import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks;
    +import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks;
    +import org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext;
    +import org.apache.flink.streaming.api.operators.StreamingRuntimeContext;
    +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
    +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState;
    +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema;
    +import org.apache.flink.util.SerializedValue;
    +
    +import org.apache.kafka.clients.consumer.ConsumerRecord;
    +import org.apache.kafka.clients.consumer.KafkaConsumer;
    +import org.apache.kafka.common.TopicPartition;
    +
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * A fetcher that fetches data from Kafka brokers via the Kafka 0.10 consumer API.
    + * 
    + * @param <T> The type of elements produced by the fetcher.
    + */
    +public class Kafka010Fetcher<T> extends Kafka09Fetcher<T> {
    +
    +	public Kafka010Fetcher(
    +			SourceContext<T> sourceContext,
    +			List<KafkaTopicPartition> assignedPartitions,
    +			SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
    +			SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
    +			StreamingRuntimeContext runtimeContext,
    +			KeyedDeserializationSchema<T> deserializer,
    +			Properties kafkaProperties,
    +			long pollTimeout,
    +			boolean useMetrics) throws Exception
    +	{
    +		super(sourceContext, assignedPartitions, watermarksPeriodic, watermarksPunctuated, runtimeContext, deserializer, kafkaProperties, pollTimeout, useMetrics);
    +	}
    +
    +	@Override
    +	protected void assignPartitionsToConsumer(KafkaConsumer<byte[], byte[]> consumer, List<TopicPartition> topicPartitions) {
    +		consumer.assign(topicPartitions);
    +	}
    +
    +	/**
    +	 * Emit record Kafka-timestamp aware.
    +	 */
    +	@Override
    +	protected <R> void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, R kafkaRecord) throws Exception {
    --- End diff --
    
    I agree that this is confusing at the `AbstractFetcher` level. I'll look into that as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75571320
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,198 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This Producer wraps a Flink Kafka 0.9 Producer, overriding only
    + * the "processElement" / "invoke" method.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> {
    --- End diff --
    
    Overall, I think the solution here seems a bit too "hacky" to me. Here's a few disadvantages I see: 1) the code in `processElement()` of this class is still quite a bit duplicate of the `invoke()` in `FlinkKafaProducerBase`. 2) with `writeToKafka()` we need to have a different usage style for the Kafka producer in 0.10, compared to previous versions.
    
    If I'm correct, I guess its due to the fact that we don't have access to the embedded record timestamp in the provided `next` in the usual `invoke`?
    
    I'm wondering whether or not a cleaner solution is to introduce a `serializeTimestamp(T element)` method in the `KeyedSerializationSchema`, and we can simply use that to make `FlinkKafkaProducerBase#invoke()` more general by replacing instantiation of `ProducerRecord`s with an abstract version-specific method. Then, `FlinkKafkaProducer010` can simply extend `FlinkKafkaProducer09`.
    
    We'll need a migration plan though if we're going to change the serialization schema interface. If we're going for it, might as well migrate the `KeyedDeserializationSchema` to include the timestamp too (give `null` for 0.8, 0.9).
    Otherwise, I think the solution here is ok for a short term solution. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @rmetzger Thanks for addressing the comments! Did a final pass, and the changes look good to me.
    I agree with merging the connector as is. Adding the timestamp to the regular sink interface seems like a good long term solution.
    
    +1 to merge once travis turns green ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Thanks Robert for addressing my comments :)
    
    Overall, I like the new hybrid producer approach. However, I'm still curious whether or not it is possible / reasonable to drop the `FlinkKafkaProducer010Configuration` return type of invocation (b), and let both invocation methods return `FlinkKafkaProducer010` instead. So,
    
    ```
    FlinkKafkaProducer010 kafka = new FlinkKafkaProducer010(...)
    // or FlinkKafkaProducer010 kafka = FlinkKafkaProducer010.writeToKafkaWithTimestamps(...) for timestamp support
    
    // setter config methods directly done on the FlinkKafkaProducer010 instance regardless of (a) or (b)
    kafka.setLogFailuresOnly(true)
    kafka.setFlushOnCheckpoint(true)
    kafka.setWriteTimestampToKafka(true) // would not have effect if original invocation method (a) was used
    ```
    
    But we'll need to be bit hacky in `invokeInternal(element, elementTimestamp)`, something like only letting the given `timestamp` to `ProducerRecord` be non-null if `writeTimestampToKafka && elementTimestamp != Long.MIN_VALUE`.
    
    What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by nemccarthy <gi...@git.apache.org>.
Github user nemccarthy commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    +1 for this pr


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75571478
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,198 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This Producer wraps a Flink Kafka 0.9 Producer, overriding only
    + * the "processElement" / "invoke" method.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> {
    --- End diff --
    
    But on the other hand, the downside for the approach above would be that users always have to explicitly extract the timestamp again (even if the record already has an embedded timestamp) to emit them to Kafka.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76268687
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,399 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.common.functions.IterationRuntimeContext;
    +import org.apache.flink.api.common.functions.RichFunction;
    +import org.apache.flink.api.common.functions.RuntimeContext;
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.functions.sink.SinkFunction;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.getPropertiesFromBrokerList;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This producer is a hybrid between a regular regular sink function (a)
    + * and a custom operator (b).
    + *
    + * For (a), the class implements the SinkFunction and RichFunction interfaces.
    + * For (b), it extends the StreamTask class.
    + *
    + * Details about approach (a):
    + *
    + *  Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the
    + *  DataStream.addSink() method.
    + *  Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record
    + *  the Kafka 0.10 producer has a section invocation option, approach (b).
    + *
    + * Details about approach (b):
    + *  Kafka 0.10 supports writing the timestamp attached to a record to Kafka. When adding the
    + *  FlinkKafkaProducer010 using the FlinkKafkaProducer010.writeToKafka() method, the Kafka producer
    + *  can access the internal record timestamp of the record and write it to Kafka.
    + *
    + * All methods and constructors in this class are marked with the approach they are needed for.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> implements SinkFunction<T>, RichFunction {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for timestamp writing ------------------
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										KeyedSerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										SerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 *  @param inStream The stream to write to Kafka
    +	 *  @param topicId The name of the target topic
    +	 *  @param serializationSchema A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages
    +	 *  @param producerConfig Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.
    +	 *  @param customPartitioner A serializable partitioner for assigning messages to Kafka partitions.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration<T> writeToKafka(DataStream<T> inStream,
    +																String topicId,
    +																KeyedSerializationSchema<T> serializationSchema,
    +																Properties producerConfig,
    +																KafkaPartitioner<T> customPartitioner) {
    +
    +		GenericTypeInfo<Object> objectTypeInfo = new GenericTypeInfo<>(Object.class);
    +		FlinkKafkaProducer010<T> kafkaProducer = new FlinkKafkaProducer010<>(topicId, serializationSchema, producerConfig, customPartitioner);
    +		SingleOutputStreamOperator<Object> transformation = inStream.transform("FlinKafkaProducer 0.10.x", objectTypeInfo, kafkaProducer);
    +		return new FlinkKafkaProducer010Configuration<>(transformation, kafkaProducer);
    +	}
    +
    +	// ---------------------- Regular constructors w/o timestamp support  ------------------
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param brokerList
    +	 *			Comma separated addresses of the brokers
    +	 * @param topicId
    +	 * 			ID of the Kafka topic.
    +	 * @param serializationSchema
    +	 * 			User defined (keyless) serialization schema.
    +	 */
    +	public FlinkKafkaProducer010(String brokerList, String topicId, SerializationSchema<T> serializationSchema) {
    +		this(topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), getPropertiesFromBrokerList(brokerList), new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param topicId
    +	 * 			ID of the Kafka topic.
    +	 * @param serializationSchema
    +	 * 			User defined (keyless) serialization schema.
    +	 * @param producerConfig
    +	 * 			Properties with the producer configuration.
    +	 */
    +	public FlinkKafkaProducer010(String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig) {
    +		this(topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param topicId The topic to write data to
    +	 * @param serializationSchema A (keyless) serializable serialization schema for turning user objects into a kafka-consumable byte[]
    +	 * @param producerConfig Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.
    +	 * @param customPartitioner A serializable partitioner for assigning messages to Kafka partitions (when passing null, we'll use Kafka's partitioner)
    +	 */
    +	public FlinkKafkaProducer010(String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig, KafkaPartitioner<T> customPartitioner) {
    +		this(topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, customPartitioner);
    +
    --- End diff --
    
    nit: unnecessary empty line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76268268
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,399 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.common.functions.IterationRuntimeContext;
    +import org.apache.flink.api.common.functions.RichFunction;
    +import org.apache.flink.api.common.functions.RuntimeContext;
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.functions.sink.SinkFunction;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.getPropertiesFromBrokerList;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This producer is a hybrid between a regular regular sink function (a)
    + * and a custom operator (b).
    + *
    + * For (a), the class implements the SinkFunction and RichFunction interfaces.
    + * For (b), it extends the StreamTask class.
    + *
    + * Details about approach (a):
    + *
    + *  Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the
    + *  DataStream.addSink() method.
    + *  Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record
    + *  the Kafka 0.10 producer has a section invocation option, approach (b).
    + *
    + * Details about approach (b):
    + *  Kafka 0.10 supports writing the timestamp attached to a record to Kafka. When adding the
    + *  FlinkKafkaProducer010 using the FlinkKafkaProducer010.writeToKafka() method, the Kafka producer
    + *  can access the internal record timestamp of the record and write it to Kafka.
    + *
    + * All methods and constructors in this class are marked with the approach they are needed for.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> implements SinkFunction<T>, RichFunction {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for timestamp writing ------------------
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										KeyedSerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    --- End diff --
    
    Now with hybrid invocation methods, would it be reasonable to name this as `writeToKafkaWithTimestamps` so that it's more meaningful and differentiable from the usual invocation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    I'm currently working on rebasing the PR and addressing the comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Just looked over this briefly.
    What struck me first is that this again uses the dirty trick of adding a dependency to "Flink Kafka 0.9" and then transitively excluding "Kafka 0.9". Is there a nicer way to solve this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @tzulitai I've addressed your comments.
    The Producer is now "hybrid": you can use it with both invocation methods.
    The AbstractFetcher now accepts a long timestamp instead of a record.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75902922
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,198 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This Producer wraps a Flink Kafka 0.9 Producer, overriding only
    + * the "processElement" / "invoke" method.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> {
    --- End diff --
    
    Exactly. The problem is that there is no way of accessing the system's event time record.
    
    Maybe I can try and build something like a hybrid producer that works with both invocation methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75569873
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka010Fetcher.java ---
    @@ -0,0 +1,91 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka.internal;
    +
    +import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks;
    +import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks;
    +import org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext;
    +import org.apache.flink.streaming.api.operators.StreamingRuntimeContext;
    +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition;
    +import org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState;
    +import org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema;
    +import org.apache.flink.util.SerializedValue;
    +
    +import org.apache.kafka.clients.consumer.ConsumerRecord;
    +import org.apache.kafka.clients.consumer.KafkaConsumer;
    +import org.apache.kafka.common.TopicPartition;
    +
    +import java.util.List;
    +import java.util.Properties;
    +
    +/**
    + * A fetcher that fetches data from Kafka brokers via the Kafka 0.10 consumer API.
    + * 
    + * @param <T> The type of elements produced by the fetcher.
    + */
    +public class Kafka010Fetcher<T> extends Kafka09Fetcher<T> {
    +
    +	public Kafka010Fetcher(
    +			SourceContext<T> sourceContext,
    +			List<KafkaTopicPartition> assignedPartitions,
    +			SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
    +			SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
    +			StreamingRuntimeContext runtimeContext,
    +			KeyedDeserializationSchema<T> deserializer,
    +			Properties kafkaProperties,
    +			long pollTimeout,
    +			boolean useMetrics) throws Exception
    +	{
    +		super(sourceContext, assignedPartitions, watermarksPeriodic, watermarksPunctuated, runtimeContext, deserializer, kafkaProperties, pollTimeout, useMetrics);
    +	}
    +
    +	@Override
    +	protected void assignPartitionsToConsumer(KafkaConsumer<byte[], byte[]> consumer, List<TopicPartition> topicPartitions) {
    +		consumer.assign(topicPartitions);
    +	}
    +
    +	/**
    +	 * Emit record Kafka-timestamp aware.
    +	 */
    +	@Override
    +	protected <R> void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, R kafkaRecord) throws Exception {
    --- End diff --
    
    Just realized here the reason for the new `kafkaRecord`.
    
    However, would it be better to add an `recordTimestamp` argument instead of passing the original Kafka record?
    If we don't have such a timestamp (for 0.8, 0.9), a `null` can be accepted. In `AbstractFetcher#emitRecord()`, we check if the value is `null` or not and correspondingly call `collect()` or `collectWithTimestamp`. The same goes for passing the `Long.MIN_VALUE` / timestamp to `emitRecordWithTimestampAndPeriodicWatermark` and `emitRecordWithTimestampAndPunctuatedWatermark `.
    
    IMHO, I think this way the new code will be more meaningful and less confusing for the base `AbstractFetcher`, and also we won't need to override `emitRecord` in 0.10.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75572212
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,198 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This Producer wraps a Flink Kafka 0.9 Producer, overriding only
    + * the "processElement" / "invoke" method.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for the producer ------------------ //
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										KeyedSerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										SerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *  @param inStream The stream to write to Kafka
    +	 *  @param topicId The name of the target topic
    +	 *  @param serializationSchema A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages
    +	 *  @param producerConfig Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.
    +	 *  @param customPartitioner A serializable partitioner for assigning messages to Kafka partitions.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration<T> writeToKafka(DataStream<T> inStream,
    +																String topicId,
    +																KeyedSerializationSchema<T> serializationSchema,
    +																Properties producerConfig,
    +																KafkaPartitioner<T> customPartitioner) {
    +		GenericTypeInfo<Object> objectTypeInfo = new GenericTypeInfo<>(Object.class);
    +		FlinkKafkaProducer010<T> kafkaProducer = new FlinkKafkaProducer010<>(topicId, serializationSchema, producerConfig, customPartitioner);
    +		SingleOutputStreamOperator<Object> transformation = inStream.transform("FlinKafkaProducer 0.10.x", objectTypeInfo, kafkaProducer);
    +		return new FlinkKafkaProducer010Configuration<>(transformation, kafkaProducer);
    +	}
    +
    +	/**
    +	 * Configuration object returned by the writeToKafka() call.
    +	 */
    +	public static class FlinkKafkaProducer010Configuration<T> extends DataStreamSink<T> {
    --- End diff --
    
    I wonder if we can move the config setter methods into `FlinkKafkaProducer010`, and let `writeToKafka` return the created 0.10 producer instead of instantiating a new `FlinkKafkaProducer010Configuration`? My reasoning is that I find it a bit strange to set these config values on a separate config object, but not "applying" it anywhere. I think users would normally expect they have to "apply" a completed config object somewhere to take effect. So, I have something like this in mind:
    
    In constructor of `FlinkKafkaProducer010`
    ```
    private FlinkKafaProducer010(...) {
      super(new FlinkKafkaProducer09<>(...));
      this.wrapped09Producer = (FlinkKafkaProducer09<T>) userFunction; // wrapped09Producer as FlinkKafkafProducer010's class field
    }
    ```
    Then FlinkKafkaProducer010 can directly have the setter methods:
    ```
    public void setFlushOnCheckpoint(boolean flush) {
        this.wrapped09producer.setFlushOnCheckpoint(flush);
    }
    
    public void setLogFailuresOnly(boolean logFailuresOnly) {
    	this.wrapped09producer.setLogFailuresOnly(logFailuresOnly);
    }
    
    public void setWriteTimestampToKafka(boolean writeTimestampToKafka) {
    	this.writeTimestampToKafka = writeTimestampToKafka;
    }
    ```
    
    Then users can use the producer like this:
    ```
    FlinkKafkaProducer010 producer = FlinkKafkaProducer010.writeToKafka(...)
    producer.setLogFailuresOnly(...)
    producer.setFlushOnCheckpoint(...)
    producer.setWriteTimestampToKafka(...)
    ```
    
    Although still a bit different from the usage patterns of previous versions, it looks a bit more similar compared to what the PR has now. I'm not sure if I may be missing something that lead to choosing to have a separate `FlinkKafkaProducer010Configuration`. Is there any particular reason?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76275805
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/SimpleConsumerThread.java ---
    @@ -376,7 +376,7 @@ else if (partitionsRemoved) {
     								continue partitionsLoop;
     							}
     							
    -							owner.emitRecord(value, currentPartition, offset, msg);
    +							owner.emitRecord(value, currentPartition, offset, Long.MAX_VALUE);
    --- End diff --
    
    I think this is supposed to give a `Long.MIN_VALUE` instead of MAX? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Thank you for the review. I'll address your comments, rebase again, test it, and if it turns green merge it ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by eliaslevy <gi...@git.apache.org>.
Github user eliaslevy commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @rmetzger that's the one. NP.  I realize breaking it up makes things easier.  I just thought I'd mention it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76277062
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/internals/AbstractFetcherTimestampsTest.java ---
    @@ -135,45 +135,45 @@ public void testPeriodicWatermarks() throws Exception {
     		// elements generate a watermark if the timestamp is a multiple of three
     
     		// elements for partition 1
    -		fetcher.emitRecord(1L, part1, 1L, new ConsumerRecord<byte[], byte[]>(testTopic, 7, new byte[]{0}, 1L));
    -		fetcher.emitRecord(2L, part1, 2L, new ConsumerRecord<byte[], byte[]>(testTopic, 7, new byte[]{0}, 2L));
    -		fetcher.emitRecord(3L, part1, 3L, new ConsumerRecord<byte[], byte[]>(testTopic, 7, new byte[]{0}, 3L));
    +		fetcher.emitRecord(1L, part1, 1L, Long.MAX_VALUE);
    +		fetcher.emitRecord(2L, part1, 2L, Long.MAX_VALUE);
    +		fetcher.emitRecord(3L, part1, 3L, Long.MAX_VALUE);
     		assertEquals(3L, sourceContext.getLatestElement().getValue().longValue());
     		assertEquals(3L, sourceContext.getLatestElement().getTimestamp());
     
     		// elements for partition 2
    -		fetcher.emitRecord(12L, part2, 1L, new ConsumerRecord<byte[], byte[]>(testTopic, 13, new byte[]{0}, 1L));
    +		fetcher.emitRecord(12L, part2, 1L, Long.MAX_VALUE);
     		assertEquals(12L, sourceContext.getLatestElement().getValue().longValue());
     		assertEquals(12L, sourceContext.getLatestElement().getTimestamp());
     
     		// elements for partition 3
    -		fetcher.emitRecord(101L, part3, 1L, new ConsumerRecord<byte[], byte[]>(testTopic, 21, new byte[]{0}, 1L));
    -		fetcher.emitRecord(102L, part3, 2L, new ConsumerRecord<byte[], byte[]>(testTopic, 21, new byte[]{0}, 2L));
    +		fetcher.emitRecord(101L, part3, 1L, Long.MAX_VALUE);
    +		fetcher.emitRecord(102L, part3, 2L, Long.MAX_VALUE);
     		assertEquals(102L, sourceContext.getLatestElement().getValue().longValue());
     		assertEquals(102L, sourceContext.getLatestElement().getTimestamp());
     
     		// now, we should have a watermark (this blocks until the periodic thread emitted the watermark)
     		assertEquals(3L, sourceContext.getLatestWatermark().getTimestamp());
     
     		// advance partition 3
    -		fetcher.emitRecord(1003L, part3, 3L, new ConsumerRecord<byte[], byte[]>(testTopic, 21, new byte[]{0}, 3L));
    -		fetcher.emitRecord(1004L, part3, 4L, new ConsumerRecord<byte[], byte[]>(testTopic, 21, new byte[]{0}, 4L));
    -		fetcher.emitRecord(1005L, part3, 5L, new ConsumerRecord<byte[], byte[]>(testTopic, 21, new byte[]{0}, 5L));
    +		fetcher.emitRecord(1003L, part3, 3L, Long.MAX_VALUE);
    +		fetcher.emitRecord(1004L, part3, 4L, Long.MAX_VALUE);
    +		fetcher.emitRecord(1005L, part3, 5L, Long.MAX_VALUE);
     		assertEquals(1005L, sourceContext.getLatestElement().getValue().longValue());
     		assertEquals(1005L, sourceContext.getLatestElement().getTimestamp());
     
     		// advance partition 1 beyond partition 2 - this bumps the watermark
    -		fetcher.emitRecord(30L, part1, 4L, new ConsumerRecord<byte[], byte[]>(testTopic, 7, new byte[]{0}, 4L));
    +		fetcher.emitRecord(30L, part1, 4L, Long.MAX_VALUE);
     		assertEquals(30L, sourceContext.getLatestElement().getValue().longValue());
     		assertEquals(30L, sourceContext.getLatestElement().getTimestamp());
     		
     		// this blocks until the periodic thread emitted the watermark
     		assertEquals(12L, sourceContext.getLatestWatermark().getTimestamp());
     
     		// advance partition 2 again - this bumps the watermark
    -		fetcher.emitRecord(13L, part2, 2L, new ConsumerRecord<byte[], byte[]>(testTopic, 13, new byte[]{0}, 2L));
    -		fetcher.emitRecord(14L, part2, 3L, new ConsumerRecord<byte[], byte[]>(testTopic, 13, new byte[]{0}, 3L));
    -		fetcher.emitRecord(15L, part2, 3L, new ConsumerRecord<byte[], byte[]>(testTopic, 13, new byte[]{0}, 3L));
    +		fetcher.emitRecord(13L, part2, 2L, Long.MAX_VALUE);
    +		fetcher.emitRecord(14L, part2, 3L, Long.MAX_VALUE);
    +		fetcher.emitRecord(15L, part2, 3L, Long.MAX_VALUE);
    --- End diff --
    
    Same here: I think these are supposed to give a `Long.MIN_VALUE` instead of MAX?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75569347
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/AbstractFetcher.java ---
    @@ -207,15 +207,14 @@ public void restoreOffsets(HashMap<KafkaTopicPartition, Long> snapshotState) {
     	// ------------------------------------------------------------------------
     
     	/**
    -	 * 
     	 * <p>Implementation Note: This method is kept brief to be JIT inlining friendly.
     	 * That makes the fast path efficient, the extended paths are called as separate methods.
    -	 * 
     	 * @param record The record to emit
     	 * @param partitionState The state of the Kafka partition from which the record was fetched
    -	 * @param offset The offset from which the record was fetched
    +	 * @param offset The offset of the record
    +	 * @param kafkaRecord The original Kafka record
     	 */
    -	protected final void emitRecord(T record, KafkaTopicPartitionState<KPH> partitionState, long offset) {
    +	protected <R> void emitRecord(T record, KafkaTopicPartitionState<KPH> partitionState, long offset, R kafkaRecord) throws Exception {
    --- End diff --
    
    Is there a reason we need to have an extra `kafkaRecord`? It doesn't seem to be used in the function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76275879
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java ---
    @@ -262,6 +262,10 @@ public void run() {
     		}
     	}
     
    +	// Kafka09Fetcher ignores the timestamp.
    +	protected void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partition, long offset, ConsumerRecord consumerRecord) throws Exception {
    +		emitRecord(record, partition, offset, Long.MAX_VALUE);
    --- End diff --
    
    Same here: I think this is supposed to give a `Long.MIN_VALUE` instead of MAX?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76264614
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,399 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.common.functions.IterationRuntimeContext;
    +import org.apache.flink.api.common.functions.RichFunction;
    +import org.apache.flink.api.common.functions.RuntimeContext;
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.functions.sink.SinkFunction;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.getPropertiesFromBrokerList;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This producer is a hybrid between a regular regular sink function (a)
    + * and a custom operator (b).
    + *
    + * For (a), the class implements the SinkFunction and RichFunction interfaces.
    + * For (b), it extends the StreamTask class.
    + *
    + * Details about approach (a):
    + *
    + *  Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the
    + *  DataStream.addSink() method.
    + *  Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record
    + *  the Kafka 0.10 producer has a section invocation option, approach (b).
    --- End diff --
    
    'section' --> second?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by eliaslevy <gi...@git.apache.org>.
Github user eliaslevy commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    This may be the wrong place to bring this up, but as you are discussing changes to the Kafka connector API, I think it is worth bring it up.  
    
    As I've pointed out elsewhere, the current connector API makes it difficult to make use of Kafka native serializer or deserializer (`org.apache.kafka.common.[Serializer, Deserializer]`), which can be configured via the Kafka client and producer configs.  
    
    The connector code assumes that `ConsummerRecord`s and `ProducerRecord`s are both parametrized as `<byte[], byte[]>`, with the Flink serdes performing the conversion to/from `byte[]`.  This makes it difficult to make use of Confluent's `KafkaAvroSerializer` and `KafkaAvroDecoder`, which make use of their [schema registry](http://docs.confluent.io/3.0.0/schema-registry/docs/serializer-formatter.html#serializer).
    
    If you are going to change the connector API, it would be good to tackle this issue at the same time to avoid future changes.  The connector should allow the type parametrization of the Kafka consumer and producer, and should make use of a pass through Flink serde by default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @StephanEwen: The explicit exclude is actually not needed, because the kafka version defined in the connector has precedence over transitive kafka versions.
    ```
    [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ flink-connector-kafka-0.10_2.10 ---
    [INFO] org.apache.flink:flink-connector-kafka-0.10_2.10:jar:1.2-SNAPSHOT
    [INFO] +- org.apache.flink:flink-connector-kafka-0.9_2.10:jar:1.2-SNAPSHOT:compile
    [INFO] |  \- org.apache.flink:flink-connector-kafka-base_2.10:jar:1.2-SNAPSHOT:compile
    [INFO] +- org.apache.kafka:kafka-clients:jar:0.10.0.1:compile
    [INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
    [INFO] |  \- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r80629075
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -83,11 +83,11 @@
     	 * @param serializationSchema User defined serialization schema supporting key/value messages
     	 * @param producerConfig Properties with the producer configuration.
     	 */
    -	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    -										String topicId,
    -										KeyedSerializationSchema<T> serializationSchema,
    -										Properties producerConfig) {
    -		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafkaWithTimestamps(DataStream<T> inStream,
    --- End diff --
    
    Add the generic type parameter `T` to `FlinkKafkaProducer010Configuration` here too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r80628475
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,398 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.common.functions.IterationRuntimeContext;
    +import org.apache.flink.api.common.functions.RichFunction;
    +import org.apache.flink.api.common.functions.RuntimeContext;
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.functions.sink.SinkFunction;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +import static org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.getPropertiesFromBrokerList;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This producer is a hybrid between a regular regular sink function (a)
    + * and a custom operator (b).
    + *
    + * For (a), the class implements the SinkFunction and RichFunction interfaces.
    + * For (b), it extends the StreamTask class.
    + *
    + * Details about approach (a):
    + *
    + *  Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the
    + *  DataStream.addSink() method.
    + *  Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record
    + *  the Kafka 0.10 producer has a second invocation option, approach (b).
    + *
    + * Details about approach (b):
    + *  Kafka 0.10 supports writing the timestamp attached to a record to Kafka. When adding the
    + *  FlinkKafkaProducer010 using the FlinkKafkaProducer010.writeToKafkaWithTimestamps() method, the Kafka producer
    + *  can access the internal record timestamp of the record and write it to Kafka.
    + *
    + * All methods and constructors in this class are marked with the approach they are needed for.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> implements SinkFunction<T>, RichFunction {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for timestamp writing ------------------
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafkaWithTimestamps(DataStream<T> inStream,
    +																					String topicId,
    +																					KeyedSerializationSchema<T> serializationSchema,
    +																					Properties producerConfig) {
    +		return writeToKafkaWithTimestamps(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafkaWithTimestamps(DataStream<T> inStream,
    +																					String topicId,
    +																					SerializationSchema<T> serializationSchema,
    +																					Properties producerConfig) {
    +		return writeToKafkaWithTimestamps(inStream, topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)
    +	 *
    +	 *  @param inStream The stream to write to Kafka
    +	 *  @param topicId The name of the target topic
    +	 *  @param serializationSchema A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages
    +	 *  @param producerConfig Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.
    +	 *  @param customPartitioner A serializable partitioner for assigning messages to Kafka partitions.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration<T> writeToKafkaWithTimestamps(DataStream<T> inStream,
    +																					   String topicId,
    +																					   KeyedSerializationSchema<T> serializationSchema,
    +																					   Properties producerConfig,
    +																					   KafkaPartitioner<T> customPartitioner) {
    --- End diff --
    
    Checkstyle failed for these few lines here, have leading spaces :P


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r80629176
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -102,11 +102,11 @@
     	 * @param serializationSchema User defined (keyless) serialization schema.
     	 * @param producerConfig Properties with the producer configuration.
     	 */
    -	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    -										String topicId,
    -										SerializationSchema<T> serializationSchema,
    -										Properties producerConfig) {
    -		return writeToKafka(inStream, topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafkaWithTimestamps(DataStream<T> inStream,
    --- End diff --
    
    Add the generic type parameter `T` to `FlinkKafkaProducer010Configuration` here too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @eliaslevy, I assume you are referring to https://issues.apache.org/jira/browse/FLINK-4050. 
    Its good that you are mentioning the issue again, so I can move it a bit up on my TODO list.
    
    I would personally prefer to first add the Kafka 0.10 module in this pull request and then resolve FLINK-4050 independently. I know that this might lead to a little bit of duplicate work on the Kafka 0.10 code, but on the other hand its easier to discuss one issue at a time :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r75902757
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer010.java ---
    @@ -0,0 +1,198 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.streaming.connectors.kafka;
    +
    +import org.apache.flink.api.java.typeutils.GenericTypeInfo;
    +import org.apache.flink.streaming.api.datastream.DataStream;
    +import org.apache.flink.streaming.api.datastream.DataStreamSink;
    +import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
    +import org.apache.flink.streaming.api.operators.StreamSink;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner;
    +import org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner;
    +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
    +import org.apache.flink.streaming.util.serialization.KeyedSerializationSchemaWrapper;
    +import org.apache.flink.streaming.util.serialization.SerializationSchema;
    +import org.apache.kafka.clients.producer.ProducerRecord;
    +
    +import java.util.Properties;
    +
    +
    +/**
    + * Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x
    + *
    + * Implementation note: This Producer wraps a Flink Kafka 0.9 Producer, overriding only
    + * the "processElement" / "invoke" method.
    + */
    +public class FlinkKafkaProducer010<T> extends StreamSink<T> {
    +
    +	/**
    +	 * Flag controlling whether we are writing the Flink record's timestamp into Kafka.
    +	 */
    +	private boolean writeTimestampToKafka = false;
    +
    +	// ---------------------- "Constructors" for the producer ------------------ //
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined serialization schema supporting key/value messages
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										KeyedSerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, serializationSchema, producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to
    +	 * the topic.
    +	 *
    +	 * @param inStream The stream to write to Kafka
    +	 * @param topicId ID of the Kafka topic.
    +	 * @param serializationSchema User defined (keyless) serialization schema.
    +	 * @param producerConfig Properties with the producer configuration.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration writeToKafka(DataStream<T> inStream,
    +										String topicId,
    +										SerializationSchema<T> serializationSchema,
    +										Properties producerConfig) {
    +		return writeToKafka(inStream, topicId, new KeyedSerializationSchemaWrapper<>(serializationSchema), producerConfig, new FixedPartitioner<T>());
    +	}
    +
    +	/**
    +	 * Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to
    +	 * the topic.
    +	 *  @param inStream The stream to write to Kafka
    +	 *  @param topicId The name of the target topic
    +	 *  @param serializationSchema A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages
    +	 *  @param producerConfig Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.
    +	 *  @param customPartitioner A serializable partitioner for assigning messages to Kafka partitions.
    +	 */
    +	public static <T> FlinkKafkaProducer010Configuration<T> writeToKafka(DataStream<T> inStream,
    +																String topicId,
    +																KeyedSerializationSchema<T> serializationSchema,
    +																Properties producerConfig,
    +																KafkaPartitioner<T> customPartitioner) {
    +		GenericTypeInfo<Object> objectTypeInfo = new GenericTypeInfo<>(Object.class);
    +		FlinkKafkaProducer010<T> kafkaProducer = new FlinkKafkaProducer010<>(topicId, serializationSchema, producerConfig, customPartitioner);
    +		SingleOutputStreamOperator<Object> transformation = inStream.transform("FlinKafkaProducer 0.10.x", objectTypeInfo, kafkaProducer);
    +		return new FlinkKafkaProducer010Configuration<>(transformation, kafkaProducer);
    +	}
    +
    +	/**
    +	 * Configuration object returned by the writeToKafka() call.
    +	 */
    +	public static class FlinkKafkaProducer010Configuration<T> extends DataStreamSink<T> {
    --- End diff --
    
    I'll update it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @cjstehno I would expect this to be in the 1.2.0 major release, which would probably be ~2 months from now according to Flink's past release cycle. The Flink community usually doesn't release major new features like this between minor bugfix releases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    @tzulitai I addressed all your comments except the one relating `FlinkKafkaProducer010Configuration`: I had a quick offline discussion with @StephanEwen about the issue and he suggested to add the timestamp to the regular sink interface.
    But I would like to make that change separate from this one, and merge the Kafka 0.10. support as-is. This will make it easier for people to try it out now and provide us with feedback. Also, I think some other Kafka related pull requests are blocked on this one.
    
    @tzulitai could you do a final pass over the changes. If you agree, I'd like to merge it afterwards.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2369: [FLINK-4035] Add a streaming connector for Apache ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2369#discussion_r76274826
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka010Fetcher.java ---
    @@ -60,12 +60,17 @@ protected void assignPartitionsToConsumer(KafkaConsumer<byte[], byte[]> consumer
     		consumer.assign(topicPartitions);
     	}
     
    +	@Override
    +	protected void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partition, long offset, ConsumerRecord consumerRecord) throws Exception {
    +		// pass timestamp
    +		super.emitRecord(record, partition, offset, consumerRecord.timestamp());
    +	}
    +
     	/**
     	 * Emit record Kafka-timestamp aware.
     	 */
     	@Override
    -	protected <R> void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, R kafkaRecord) throws Exception {
    -		long timestamp = ((ConsumerRecord) kafkaRecord).timestamp();
    +	protected void emitRecord(T record, KafkaTopicPartitionState<TopicPartition> partitionState, long offset, long timestamp) throws Exception {
     		if (timestampWatermarkMode == NO_TIMESTAMPS_WATERMARKS) {
    --- End diff --
    
    Is it possible to let `AbstractFetcher#emitRecord` determine whether to call `collectWithTimestamp` or `collect` depending on the provided timestamp (== `Long.MIN_VALUE`)? Then, we won't need to override the base `emitRecord` here, correct?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Tests are running https://travis-ci.org/rmetzger/flink/builds/166441047


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Just found some minor issues that can be fixed when merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2369: [FLINK-4035] Add a streaming connector for Apache Kafka 0...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2369
  
    Thank you for the review @tzulitai. I'll try to find some time soon to look into your comments in detail.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---