You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/25 15:20:32 UTC

[GitHub] [pinot] ddcprg opened a new pull request, #8774: Make row metadata available in Kafka 2.0 record ingestion

ddcprg opened a new pull request, #8774:
URL: https://github.com/apache/pinot/pull/8774

   This is a new feature Kafka 2.0 stream ingestion plugin
   
   Fixes #8773 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882472455


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   thank you @xiangfu0 @KKcorps  for reviewing, would this be generic enough to move it to `StreamConfigProperties` if I renamed it `consumption.metadata.populate` or something similar? I've reviewed both Pulsar and Kinesis stream plugins and this would apply to these 2 as well. Then, I think the property can be actually documented on the page that @KKcorps has mentioned above if we do this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882472455


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   thank you @xiangfu0 @KKcorps  for reviewing, would this be generic enough to move it to `StreamConfigProperties` if I renamed it `consumption.metadata.populate` or something similar? I've reviewed both Pulsar and Kinesis stream plugins and this would apply to these 2 as well. Then, I think the property can be actually documented on the page that @KKcorps has mentioned above if we do this. I can extend these changes to Pulsar and Kinesis in a separate PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
navina commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882094320


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {

Review Comment:
   Iiuc, you are trying to replace `StreamMessageMetadata` with a kafka-specific metadata that takes into account the record timestamp as opposed to the pinot's ingestion time. please correct me if I am mistaken! 



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {

Review Comment:
   maybe some javadocs might help! 



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   +1 to the above suggestion. 
   You are processing a record and not really creating an new instance of anything. So the use of factory pattern is confusing. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883183322


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {
+
+  private final long _ingestionTimeStamp;
+
+  KafkaMessageMetadata(long ingestionTimeStamp) {
+    _ingestionTimeStamp = ingestionTimeStamp;
+  }
+
+  @Override
+  public long getIngestionTimeMs() {

Review Comment:
   please keep the name same across getter and field (_ingestionTimeMillis and getIngestionTimeMillis())



##########
pinot-spi/src/main/java/org/apache/pinot/spi/stream/StreamConfigProperties.java:
##########
@@ -49,6 +49,7 @@ private StreamConfigProperties() {
   public static final String GROUP_ID = "hlc.group.id";
   public static final String PARTITION_MSG_OFFSET_FACTORY_CLASS = "partition.offset.factory.class.name";
   public static final String TOPIC_CONSUMPTION_RATE_LIMIT = "topic.consumption.rate.limit";
+  public static final String TOPIC_CONSUMPTION_METADATA_POPULATE = "topic.consumption.metadata.populate";

Review Comment:
   lets keep it simple and call it "populate.metadata", so the property will read "stream.kafka.populate.metadata"



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());

Review Comment:
   curious, what happens if this hasn't been explicitly populated on the producer side? what's the default value?



##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {
+
+  private final long _ingestionTimeStamp;
+
+  KafkaMessageMetadata(long ingestionTimeStamp) {
+    _ingestionTimeStamp = ingestionTimeStamp;

Review Comment:
   should this be called ingestionTimestamp or recordTimestamp/eventTimestamp/producerTimestamp ? ingestionTimestamp gives the impression that this is the time when Pinot ingested the record



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883378726


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {

Review Comment:
   Good spot, I didn't see this class to be honest. I'll get rid of this and use `StreamMessageMetadata`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882536245


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   I've pushed a commit with this change, can revert/modify in any case will squash before merging



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882472455


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   thank you @xiangfu0 @KKcorps  for reviewing, would this be generic enough to move it to `StreamConfigProperties` if I renamed it `consumption.metadata.populate` or something similar? I've reviewed both Pulsar and Kinesis stream plugins and this would apply to these 2 as well. Then, I think the property can be actually documented on the page that @KKcorps has mentioned above if we do this. I can propagate this changes to Pulsar and Kinesis in a separate PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883796352


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   > the evaluation result will be the same for every record this would be inefficient
   
   That's actually not the case. The build function is called only once in the constructor of `KafkaPartitionLevelConnectionHandler`. There's no condition in the extract method during processing of each record.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883376889


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   Thanks @sajjad-moradi @navina for your feedback. One on the reason why I've implemented it like this is precisely to avoid this conditional because this config driven and the evaluation result will be the same for every record this would be inefficient. The other reason is right now the metadata includes only 1 property but more may be added later and the construction of `KafkaMessageMetadata` may become more complex. Would it help if I rename the classes to remove the word Factory? I can replace with Extractor or Creator or Provider



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883956426


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   Oh, sorry I didn't pay attention to the example, you are actually returning a lambda. Yes, in this case you are right 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883130356


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   This is a little bit confusing as the interface MessageMetadataFactory is specific to kafka (ConsumerRecord argument of the method belongs to kafka) and then the class KafkaMessageMetadataFactory implements that!
   Since the logic is simple, I would suggest consolidating three classes `MessageMetadataFactory`, `KafkaMessageMetadataFactory`, and `NoopMessageMetadataFactory` into one. Something like the following (I used extractor suffix instead of factory):
   ```java
   public interface KafkaMessageMetadataExtractor {
   
     KafkaMessageMetadata extract(ConsumerRecord<?, ?> consumerRecord);
   
     static KafkaMessageMetadataExtractor build(boolean populateMetadata) {
       if (populateMetadata) {
         return record -> new KafkaMessageMetadata(record.timestamp());
       } else {
         return record -> null;
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883463761


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());

Review Comment:
   On the client side, the way this works is the programmer can provide a timestamp at `ProducerRecord` creation time. If this is not provided at creation time Kafka client will set the timestamp before sending the message to the broker. 
   On the broker side, the timestamp that Kafka will attach to the message will depend on how the topic is configured. If the topic is configured to use "create time" then the broker uses the timestamp set in the client side (this is the default behaviour); if the topic is configured to use "log append time" then the broker sets the message timestamp when the message is persisted.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r884124667


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   Change made



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar merged pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
npawar merged PR #8774:
URL: https://github.com/apache/pinot/pull/8774


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] KKcorps commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
KKcorps commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882406112


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   Thanks a lot for the contribution!
   We can document it here - https://docs.pinot.apache.org/configuration-reference/table. Also I think you just need to keep the property name as `populateRowMetadata`. The prefix is handled by pinot and can be changed in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on PR #8774:
URL: https://github.com/apache/pinot/pull/8774#issuecomment-1139807522

   @npawar @navina thanks for your feedback, I've made some changes as per your comments, please take one more look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883956426


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   Oh, didn't pay attention to the example, you are actually returning a lambda. Yes, in this case you are right 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r881891432


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   I've raised #8775 in relation this question



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8774:
URL: https://github.com/apache/pinot/pull/8774#issuecomment-1139751541

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8774](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8a3b78d) into [master](https://codecov.io/gh/apache/pinot/commit/963d4483db97598919fc514609496d9ecd6667d2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (963d448) will **decrease** coverage by `6.82%`.
   > The diff coverage is `70.58%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #8774      +/-   ##
   ============================================
   - Coverage     69.69%   62.86%   -6.83%     
   + Complexity     4617     4601      -16     
   ============================================
     Files          1736     1693      -43     
     Lines         91180    89224    -1956     
     Branches      13632    13411     -221     
   ============================================
   - Hits          63548    56092    -7456     
   - Misses        23206    29084    +5878     
   + Partials       4426     4048     -378     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `66.16% <ø> (+<0.01%)` | :arrow_up: |
   | unittests2 | `14.19% <70.58%> (+0.06%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pache/pinot/spi/stream/StreamConfigProperties.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvc3RyZWFtL1N0cmVhbUNvbmZpZ1Byb3BlcnRpZXMuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [...tream/kafka20/KafkaPartitionLevelStreamConfig.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxTdHJlYW1Db25maWcuamF2YQ==) | `63.15% <56.25%> (-1.85%)` | :arrow_down: |
   | [...in/stream/kafka20/MessageAndOffsetAndMetadata.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL01lc3NhZ2VBbmRPZmZzZXRBbmRNZXRhZGF0YS5qYXZh) | `57.14% <57.14%> (ø)` | |
   | [...pinot/plugin/stream/kafka20/KafkaMessageBatch.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthTWVzc2FnZUJhdGNoLmphdmE=) | `57.14% <100.00%> (-35.17%)` | :arrow_down: |
   | [.../kafka20/KafkaPartitionLevelConnectionHandler.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxDb25uZWN0aW9uSGFuZGxlci5qYXZh) | `80.95% <100.00%> (-19.05%)` | :arrow_down: |
   | [...in/stream/kafka20/KafkaPartitionLevelConsumer.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL0thZmthUGFydGl0aW9uTGV2ZWxDb25zdW1lci5qYXZh) | `64.28% <100.00%> (-2.39%)` | :arrow_down: |
   | [...am/kafka20/metadata/KafkaRowMetadataExtractor.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL21ldGFkYXRhL0thZmthUm93TWV0YWRhdGFFeHRyYWN0b3IuamF2YQ==) | `100.00% <100.00%> (ø)` | |
   | [...eam/kafka20/metadata/NoopRowMetadataExtractor.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL21ldGFkYXRhL05vb3BSb3dNZXRhZGF0YUV4dHJhY3Rvci5qYXZh) | `100.00% <100.00%> (ø)` | |
   | [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [430 more](https://codecov.io/gh/apache/pinot/pull/8774/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [963d448...8a3b78d](https://codecov.io/gh/apache/pinot/pull/8774?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
xiangfu0 commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882436187


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   Shall we move this static to KafkaStreamConfigProperties?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r881882893


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   I think this should be documented somewhere, not sure where the best place is



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on PR #8774:
URL: https://github.com/apache/pinot/pull/8774#issuecomment-1140782960

   Than you Neha


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883793864


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());
+  }
+}

Review Comment:
   I've replace *Factory with *Extractor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r882472455


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelStreamConfig.java:
##########
@@ -31,13 +31,16 @@
  */
 public class KafkaPartitionLevelStreamConfig {
 
+  public static final String POPULATE_ROW_METADATA = "stream.kafka.populateRowMetadata";

Review Comment:
   thank you @xiangfu0 @KKcorps  for reviewing, would this be generic enough to move it to `StreamConfigProperties` if I renamed it `consumption.metadata.populate` or something similar? I've reviewed both Pulsar and Kinesis stream plugins and this would apply to these 2 as well. The I think the property can be actually documented on the page that @KKcorps has mentioned above if we do this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883463761


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadataFactory.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+
+public class KafkaMessageMetadataFactory implements MessageMetadataFactory {
+  @Override
+  public KafkaMessageMetadata create(ConsumerRecord<?, ?> consumerRecord) {
+    return new KafkaMessageMetadata(consumerRecord.timestamp());

Review Comment:
   On the client side, the way this works is the programmer can provide a timestamp at `ProducerRecord` creation time. If this is not provided at creation time Kafka client will set the timestamp before sending the message to the broker. 
   On the broker side, the timestamp that Kafka will attach to the message will depend on how the topic is configured. If the topic is configure to use "create time" the the broker uses the timestamp set in the client side (this is the default behaviour); if the topic is configured to use "log append time" the the broker set the message timestamp when the message is persisted.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ddcprg commented on a diff in pull request #8774: Make row metadata available in Kafka 2.0 record ingestion

Posted by GitBox <gi...@apache.org>.
ddcprg commented on code in PR #8774:
URL: https://github.com/apache/pinot/pull/8774#discussion_r883383602


##########
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/metadata/KafkaMessageMetadata.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.kafka20.metadata;
+
+import org.apache.pinot.spi.stream.RowMetadata;
+
+
+public class KafkaMessageMetadata implements RowMetadata {
+
+  private final long _ingestionTimeStamp;
+
+  KafkaMessageMetadata(long ingestionTimeStamp) {
+    _ingestionTimeStamp = ingestionTimeStamp;

Review Comment:
   I'll get rid of this class and use `StreamMessageMetadata` in the SPI instead



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org