You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/03/02 03:49:05 UTC

[GitHub] [pinot] 0902horn opened a new pull request #8274: Rocketmq integration

0902horn opened a new pull request #8274:
URL: https://github.com/apache/pinot/pull/8274


   ## Description
   <!-- Add a description of your PR here.
   A good description should include pointers to an issue or design document, etc.
   -->
   Add Apache RocketMQ Connector for Pinot realtime ingestion #8250
   
   
   RocketMQConsumerFactory provides StreamLevelConsumer only. RocketMQ has MessageQueue which is used like partionId in Kafka. Pinot assumes partionId as an integer, so PartitionLevelConsumer for RocketMQ is not implemented.
   
   Sample table config:
   
   {
     "tableName": "rocketmqTest",
     "tableType": "REALTIME",
     "segmentsConfig": {
       "timeColumnName": "DaysSinceEpoch",
       "schemaName": "airlineStats",
       "replicasPerPartition": "1"
     },
     "tenants": {},
     "tableIndexConfig": {
       "loadMode": "MMAP",
       "streamConfigs": {
         "streamType": "rocketmq",
         "stream.rocketmq.nameserver.list": "127.0.0.1:40018",
         "stream.rocketmq.consumer.type": "highlevel",
         "stream.rocketmq.topic.name": "rocketmq-test",
         "stream.rocketmq.consumer.prop.namespace": "",
         "stream.rocketmq.consumer.prop.auto.offset.reset": "smallest",
         "stream.rocketmq.fetch.timeout.millis": "10000",
         "stream.rocketmq.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
         "stream.rocketmq.consumer.factory.class.name": "org.apache.pinot.plugin.stream.rocketmq.RocketMQConsumerFactory",
         "realtime.segment.flush.threshold.size": "10000",
         "realtime.segment.flush.threshold.time": "1h"
       }
     },
     "metadata": {
       "customConfigs": {}
     }
   }
   ## Upgrade Notes
   Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
   * [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR fix a zero-downtime upgrade introduced earlier?
   * [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR otherwise need attention when creating release notes? Things to consider:
   - New configuration options
   - Deprecation of configurations
   - Signature changes to public methods/interfaces
   - New plugins added or old plugins removed
   * [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
   ## Release Notes
   <!-- If you have tagged this as either backward-incompat or release-notes,
   you MUST add text here that you would like to see appear in release notes of the
   next release. -->
   Apache RocketMQ connector
   <!-- If you have a series of commits adding or enabling a feature, then
   add this section only in final commit that marks the feature completed.
   Refer to earlier release notes to see examples of text.
   -->
   ## Documentation
   <!-- If you have introduced a new feature or configuration, please add it to the documentation as well.
   See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
   -->
   Coming very soon
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1056808787


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8274](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8c1f857) into [master](https://codecov.io/gh/apache/pinot/commit/c77c2c3e034d0cbe107142f62a2c89d29c566dbb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c77c2c3) will **decrease** coverage by `41.89%`.
   > The diff coverage is `83.33%`.
   
   > :exclamation: Current head 8c1f857 differs from pull request most recent head 0df2ebc. Consider uploading reports for the commit 0df2ebc to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/8274/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #8274       +/-   ##
   =============================================
   - Coverage     70.80%   28.90%   -41.90%     
   =============================================
     Files          1631     1619       -12     
     Lines         85449    85107      -342     
     Branches      12878    12839       -39     
   =============================================
   - Hits          60501    24602    -35899     
   - Misses        20757    58241    +37484     
   + Partials       4191     2264     -1927     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `28.90% <83.33%> (-0.04%)` | :arrow_down: |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../org/apache/pinot/core/common/MinionConstants.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vTWluaW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <ø> (ø)` | |
   | [.../spi/creator/name/SegmentNameGeneratorFactory.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvbmFtZS9TZWdtZW50TmFtZUdlbmVyYXRvckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (-57.90%)` | :arrow_down: |
   | [...rocessing/framework/SegmentProcessorFramework.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRQcm9jZXNzb3JGcmFtZXdvcmsuamF2YQ==) | `89.85% <33.33%> (-8.68%)` | :arrow_down: |
   | [...ot/common/request/context/predicate/Predicate.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVxdWVzdC9jb250ZXh0L3ByZWRpY2F0ZS9QcmVkaWNhdGUuamF2YQ==) | `100.00% <100.00%> (+33.33%)` | :arrow_up: |
   | [...re/segment/processing/framework/SegmentConfig.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRDb25maWcuamF2YQ==) | `87.50% <100.00%> (-2.50%)` | :arrow_down: |
   | [...ache/pinot/plugin/minion/tasks/MergeTaskUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvTWVyZ2VUYXNrVXRpbHMuamF2YQ==) | `69.23% <100.00%> (-26.85%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1179 more](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c77c2c3...0df2ebc](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r820417212



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQStreamLevelConsumer.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamDecoderProvider;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.rocketmq.client.consumer.DefaultLitePullConsumer;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * A {@link StreamLevelConsumer} implementation for the RocketMQ stream
+ */
+public class RocketMQStreamLevelConsumer implements StreamLevelConsumer {
+
+  private StreamMessageDecoder _messageDecoder;
+  private Logger _instanceLogger;
+
+  private StreamConfig _streamConfig;
+  private RocketMQConfig _rocketmqStreamLevelStreamConfig;
+
+  private DefaultLitePullConsumer _consumer;
+  private List<MessageExt> _consumerRecords;
+  private Iterator<MessageExt> _rocketmqIterator;
+  private Map<MessageQueue, Long> _consumerOffsets = new HashMap<>(); // tracking current consumed records offsets.
+
+  private long _lastLogTime = 0;
+  private long _lastCount = 0;
+  private long _currentCount = 0L;
+
+  public RocketMQStreamLevelConsumer(String clientId, String tableName, StreamConfig streamConfig,
+      Set<String> sourceFields, String groupId) {
+    _streamConfig = streamConfig;
+    _rocketmqStreamLevelStreamConfig = new RocketMQConfig(streamConfig, groupId);
+
+    _messageDecoder = StreamDecoderProvider.create(streamConfig, sourceFields);
+
+    _instanceLogger =
+        LoggerFactory.getLogger(RocketMQConfig.class.getName() + "_" + tableName + "_" + streamConfig.getTopicName());
+    _instanceLogger.info("RocketMQStreamLevelConsumer: streamConfig : {}", _streamConfig);
+  }
+
+  @Override
+  public void start()
+      throws Exception {
+    _consumer = RocketMQStreamLevelConsumerManager.acquireRocketMQConsumerForConfig(_rocketmqStreamLevelStreamConfig);
+  }
+
+  private void updateRocketMQIterator() {
+    _consumerRecords = _consumer.poll(_streamConfig.getFetchTimeoutMillis());
+    _rocketmqIterator = _consumerRecords.iterator();
+  }
+
+  private void resetOffsets() {
+    for (MessageQueue queue : _consumerOffsets.keySet()) {
+      long offsetToSeek = _consumerOffsets.get(queue);
+      try {
+        _consumer.seek(queue, offsetToSeek);
+      } catch (MQClientException e) {
+        _instanceLogger.warn("Caught exception while seek offset", e);
+      }
+    }
+  }
+
+  @Override
+  public GenericRow next(GenericRow destination) {
+    if (_rocketmqIterator == null || !_rocketmqIterator.hasNext()) {
+      updateRocketMQIterator();
+    }
+    if (_rocketmqIterator.hasNext()) {
+      try {
+        final MessageExt record = _rocketmqIterator.next();
+        updateOffsets(new MessageQueue(record.getTopic(), record.getBrokerName(), record.getQueueId()),
+            record.getQueueOffset());
+        destination = _messageDecoder.decode(record.getBody(), destination);
+
+        _currentCount++;
+
+        final long now = System.currentTimeMillis();
+        // Log every minute or 100k events
+        if (now - _lastLogTime > 60000 || _currentCount - _lastCount >= 100000) {
+          if (_lastCount == 0) {
+            _instanceLogger
+                .info("Consumed {} events from rocketmq stream {}", _currentCount, _streamConfig.getTopicName());
+          } else {
+            _instanceLogger.info("Consumed {} events from rocketmq stream {} (rate:{}/s)", _currentCount - _lastCount,
+                _streamConfig.getTopicName(), (float) (_currentCount - _lastCount) * 1000 / (now - _lastLogTime));
+          }
+          _lastCount = _currentCount;
+          _lastLogTime = now;
+        }
+        return destination;
+      } catch (Exception e) {
+        _instanceLogger.warn("Caught exception while consuming events", e);
+        throw e;
+      }
+    }
+    return null;
+  }
+
+  private void updateOffsets(MessageQueue queue, long offset) {

Review comment:
       > Does the consumer maintain a queue per broker per topic ?
   
   In RocketMQ, for example, a topic T may have 8 queues in a cluster of 4 brokers, so each broker has 2 queues for topic T.
   
   > In RocketMQ, is the message ordering guaranteed for messages within the same broker?
   
   The message ordering is guaranteed within the same queue.  A broker may have multiple queues for a topic.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn edited a comment on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn edited a comment on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1060163126


   Hi @navina,
   
   > Is the `partitionId` being an `int` in Pinot the only reason for implementing this connector as a `StreamLevelConsumer` ? 
   
   Yes. 
   
   > How is it defined in RocketMQ?
   
   MessageQueue is defined as a  3-tuple like *(topic, broker, queueId)*. Notice that *queueId* is a number in the broker scoped.
   
   > Does RocketMQ support both partition-level consumer and stream-level consumer models?
   
   Yes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r820412903



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQStreamLevelConsumer.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamDecoderProvider;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.rocketmq.client.consumer.DefaultLitePullConsumer;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * A {@link StreamLevelConsumer} implementation for the RocketMQ stream
+ */
+public class RocketMQStreamLevelConsumer implements StreamLevelConsumer {
+
+  private StreamMessageDecoder _messageDecoder;
+  private Logger _instanceLogger;
+
+  private StreamConfig _streamConfig;
+  private RocketMQConfig _rocketmqStreamLevelStreamConfig;
+
+  private DefaultLitePullConsumer _consumer;
+  private List<MessageExt> _consumerRecords;
+  private Iterator<MessageExt> _rocketmqIterator;
+  private Map<MessageQueue, Long> _consumerOffsets = new HashMap<>(); // tracking current consumed records offsets.
+
+  private long _lastLogTime = 0;
+  private long _lastCount = 0;
+  private long _currentCount = 0L;
+
+  public RocketMQStreamLevelConsumer(String clientId, String tableName, StreamConfig streamConfig,
+      Set<String> sourceFields, String groupId) {
+    _streamConfig = streamConfig;
+    _rocketmqStreamLevelStreamConfig = new RocketMQConfig(streamConfig, groupId);
+
+    _messageDecoder = StreamDecoderProvider.create(streamConfig, sourceFields);
+
+    _instanceLogger =
+        LoggerFactory.getLogger(RocketMQConfig.class.getName() + "_" + tableName + "_" + streamConfig.getTopicName());
+    _instanceLogger.info("RocketMQStreamLevelConsumer: streamConfig : {}", _streamConfig);
+  }
+
+  @Override
+  public void start()
+      throws Exception {
+    _consumer = RocketMQStreamLevelConsumerManager.acquireRocketMQConsumerForConfig(_rocketmqStreamLevelStreamConfig);
+  }
+
+  private void updateRocketMQIterator() {
+    _consumerRecords = _consumer.poll(_streamConfig.getFetchTimeoutMillis());
+    _rocketmqIterator = _consumerRecords.iterator();
+  }
+
+  private void resetOffsets() {
+    for (MessageQueue queue : _consumerOffsets.keySet()) {
+      long offsetToSeek = _consumerOffsets.get(queue);
+      try {
+        _consumer.seek(queue, offsetToSeek);
+      } catch (MQClientException e) {
+        _instanceLogger.warn("Caught exception while seek offset", e);
+      }
+    }
+  }
+
+  @Override
+  public GenericRow next(GenericRow destination) {
+    if (_rocketmqIterator == null || !_rocketmqIterator.hasNext()) {
+      updateRocketMQIterator();
+    }
+    if (_rocketmqIterator.hasNext()) {
+      try {
+        final MessageExt record = _rocketmqIterator.next();
+        updateOffsets(new MessageQueue(record.getTopic(), record.getBrokerName(), record.getQueueId()),
+            record.getQueueOffset());
+        destination = _messageDecoder.decode(record.getBody(), destination);
+
+        _currentCount++;
+
+        final long now = System.currentTimeMillis();
+        // Log every minute or 100k events
+        if (now - _lastLogTime > 60000 || _currentCount - _lastCount >= 100000) {
+          if (_lastCount == 0) {
+            _instanceLogger
+                .info("Consumed {} events from rocketmq stream {}", _currentCount, _streamConfig.getTopicName());
+          } else {
+            _instanceLogger.info("Consumed {} events from rocketmq stream {} (rate:{}/s)", _currentCount - _lastCount,
+                _streamConfig.getTopicName(), (float) (_currentCount - _lastCount) * 1000 / (now - _lastLogTime));
+          }
+          _lastCount = _currentCount;
+          _lastLogTime = now;
+        }
+        return destination;
+      } catch (Exception e) {
+        _instanceLogger.warn("Caught exception while consuming events", e);
+        throw e;
+      }
+    }
+    return null;
+  }
+
+  private void updateOffsets(MessageQueue queue, long offset) {
+    _consumerOffsets.put(queue, offset + 1);

Review comment:
       Actually, it is used in [resetOffsets@82](https://github.com/apache/pinot/pull/8274/files/0df2ebcfd8891dbb7a1dbeca0f8fd5556e969ff5#diff-1ccff2998469b33edf9814080c3edcfe3bcd733189441a027ea94222246d1574R82). 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn commented on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn commented on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1060164469


   @richardstartin @navina @npawar  Thank you for detailed comments! 
   Working on this today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn commented on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn commented on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1060163126


   > Is the `partitionId` being an `int` in Pinot the only reason for implementing this connector as a `StreamLevelConsumer` ? 
   Yes. 
   
   > How is it defined in RocketMQ?
   MessageQueue is defined as a  3-tuple like *(topic, broker, queueId)*. Notice that *queueId* is a number in the broker scoped.
   
   > Does RocketMQ support both partition-level consumer and stream-level consumer models?
   Yes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r820520408



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQConsumerFactory.java
##########
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.Set;
+import org.apache.pinot.spi.stream.PartitionLevelConsumer;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+
+
+/**
+ * A {@link StreamConsumerFactory} implementation for the RocketMQ stream
+ */
+public class RocketMQConsumerFactory extends StreamConsumerFactory {
+  @Override
+  public PartitionLevelConsumer createPartitionLevelConsumer(String clientId, int partition) {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public StreamLevelConsumer createStreamLevelConsumer(String clientId, String tableName, Set<String> fieldsToRead,
+      String groupId) {
+    return new RocketMQStreamLevelConsumer(clientId, tableName, _streamConfig, fieldsToRead, groupId);

Review comment:
       RocketMQ supports both stream level consuming and partition level consuming. The only problem is that the queues  in RocketMQ (equivalent to partitions in Kafka)  are identified by a 3-tuple like MessageQueue(topic, broker, queueId). It is difficult to implement partition level consumer of RocketMQ since Pinot assumes partitionId as integer.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] npawar commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
npawar commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r820308280



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/pom.xml
##########
@@ -0,0 +1,77 @@
+<?xml version="1.0"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <artifactId>pinot-stream-ingestion</artifactId>
+    <groupId>org.apache.pinot</groupId>
+    <version>0.10.0-SNAPSHOT</version>
+    <relativePath>..</relativePath>
+  </parent>
+
+  <artifactId>pinot-rocketmq</artifactId>
+  <name>Pinot Kinesis</name>

Review comment:
       change name to Pinot RocketMQ

##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQConsumerFactory.java
##########
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.Set;
+import org.apache.pinot.spi.stream.PartitionLevelConsumer;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+
+
+/**
+ * A {@link StreamConsumerFactory} implementation for the RocketMQ stream
+ */
+public class RocketMQConsumerFactory extends StreamConsumerFactory {
+  @Override
+  public PartitionLevelConsumer createPartitionLevelConsumer(String clientId, int partition) {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public StreamLevelConsumer createStreamLevelConsumer(String clientId, String tableName, Set<String> fieldsToRead,
+      String groupId) {
+    return new RocketMQStreamLevelConsumer(clientId, tableName, _streamConfig, fieldsToRead, groupId);

Review comment:
       high level consumer is deprecated.. is there no concept of partitions in RocketMQ? or an API with partition like semantics? 

##########
File path: pinot-integration-tests/src/test/resources/rocketmq/airlineStats_data_reduced.schema
##########
@@ -0,0 +1,32 @@
+{
+  "dimensionFieldSpecs": [
+    {
+      "dataType": "INT",
+      "singleValueField": true,
+      "name": "Quarter"
+    },
+    {
+      "dataType": "STRING",
+      "singleValueField": true,
+      "name": "Origin"
+    },
+    {
+      "dataType": "STRING",
+      "singleValueField": true,
+      "name": "Destination"
+    },
+    {
+      "dataType": "INT",
+      "singleValueField": true,
+      "name": "FlightNum"
+    }
+  ],
+  "timeFieldSpec": {

Review comment:
       use dateTimeFieldSpec, as timeFieldSpec is deprecated. Folks often look at integration tests and quickstarts to copy configs, and we don't want anyone using this anymore




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn edited a comment on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn edited a comment on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1060163126


   Hi @navina,
   
   > Is the `partitionId` being an `int` in Pinot the only reason for implementing this connector as a `StreamLevelConsumer` ? 
   
   Yes. 
   
   > How is it defined in RocketMQ?
   
   MessageQueue is defined as a  3-tuple like *(topic, broker, queueId)*. Notice that *queueId* is a number in the broker scope.
   
   > Does RocketMQ support both partition-level consumer and stream-level consumer models?
   
   Yes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1056808787


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8274](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8c1f857) into [master](https://codecov.io/gh/apache/pinot/commit/c77c2c3e034d0cbe107142f62a2c89d29c566dbb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c77c2c3) will **decrease** coverage by `40.06%`.
   > The diff coverage is `83.33%`.
   
   > :exclamation: Current head 8c1f857 differs from pull request most recent head 0df2ebc. Consider uploading reports for the commit 0df2ebc to get more accurate results
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/8274/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #8274       +/-   ##
   =============================================
   - Coverage     70.80%   30.73%   -40.07%     
   =============================================
     Files          1631     1619       -12     
     Lines         85449    85107      -342     
     Branches      12878    12839       -39     
   =============================================
   - Hits          60501    26157    -34344     
   - Misses        20757    56603    +35846     
   + Partials       4191     2347     -1844     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `28.90% <83.33%> (-0.04%)` | :arrow_down: |
   | integration2 | `27.65% <83.33%> (-0.03%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../org/apache/pinot/core/common/MinionConstants.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vTWluaW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <ø> (ø)` | |
   | [.../spi/creator/name/SegmentNameGeneratorFactory.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvbmFtZS9TZWdtZW50TmFtZUdlbmVyYXRvckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (-57.90%)` | :arrow_down: |
   | [...rocessing/framework/SegmentProcessorFramework.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRQcm9jZXNzb3JGcmFtZXdvcmsuamF2YQ==) | `89.85% <33.33%> (-8.68%)` | :arrow_down: |
   | [...ot/common/request/context/predicate/Predicate.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVxdWVzdC9jb250ZXh0L3ByZWRpY2F0ZS9QcmVkaWNhdGUuamF2YQ==) | `100.00% <100.00%> (+33.33%)` | :arrow_up: |
   | [...re/segment/processing/framework/SegmentConfig.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3Byb2Nlc3NpbmcvZnJhbWV3b3JrL1NlZ21lbnRDb25maWcuamF2YQ==) | `87.50% <100.00%> (-2.50%)` | :arrow_down: |
   | [...ache/pinot/plugin/minion/tasks/MergeTaskUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvTWVyZ2VUYXNrVXRpbHMuamF2YQ==) | `69.23% <100.00%> (-26.85%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1125 more](https://codecov.io/gh/apache/pinot/pull/8274/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c77c2c3...0df2ebc](https://codecov.io/gh/apache/pinot/pull/8274?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r817606771



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/test/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQStreamLevelConsumerTest.java
##########
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.pinot.plugin.stream.rocketmq.utils.MiniRocketMQCluster;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.stream.LongMsgOffset;
+import org.apache.pinot.spi.stream.OffsetCriteria;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamConsumerFactoryProvider;
+import org.apache.pinot.spi.stream.StreamDecoderProvider;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.rocketmq.client.producer.DefaultMQProducer;
+import org.apache.rocketmq.client.producer.MessageQueueSelector;
+import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
+import org.apache.rocketmq.common.message.Message;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class RocketMQStreamLevelConsumerTest {
+
+  private static final long STABILIZE_SLEEP_DELAYS = 3000;
+  private static final String TEST_TOPIC_1 = "foo";
+  private static final String TEST_TOPIC_2 = "bar";
+  private static final int NUM_MSG_PRODUCED_PER_PARTITION = 1000;
+
+  private MiniRocketMQCluster _rocketmqCluster;
+  private String _namesrvAddress;
+
+  @BeforeClass
+  public void setUp()
+      throws Exception {
+    _rocketmqCluster = new MiniRocketMQCluster("default_cluster", "default_broker");
+    _rocketmqCluster.start();
+    _namesrvAddress = _rocketmqCluster.getNamesrvAddress();
+    System.out.println("nameserver: " + _namesrvAddress);
+    Thread.sleep(STABILIZE_SLEEP_DELAYS);
+    _rocketmqCluster.initTopic(TEST_TOPIC_1, 1);
+    _rocketmqCluster.initTopic(TEST_TOPIC_2, 2);
+    produceMsgToRocketMQ();
+    Thread.sleep(STABILIZE_SLEEP_DELAYS);
+  }
+
+  private void produceMsgToRocketMQ() {
+    DefaultMQProducer producer = new DefaultMQProducer("TEST_PRODUCER");
+    producer.setNamesrvAddr(_namesrvAddress);
+    try {
+      producer.start();
+
+      MessageQueueSelector queueSelector = (mqs, msg, arg) -> {
+        return mqs.stream().filter(q -> arg.equals(q.getQueueId())).collect(Collectors.toList()).get(0);
+      };
+
+      for (int i = 0; i < NUM_MSG_PRODUCED_PER_PARTITION; i++) {
+        producer.send(new Message(TEST_TOPIC_1, ("sample_msg_" + i).getBytes(StandardCharsets.UTF_8)));
+        // TEST_TOPIC_2 has 2 partitions
+        producer.send(new Message(TEST_TOPIC_2, ("sample_msg_0_" + i).getBytes(StandardCharsets.UTF_8)), queueSelector,
+            0);
+        producer.send(new Message(TEST_TOPIC_2, ("sample_msg_1_" + i).getBytes(StandardCharsets.UTF_8)), queueSelector,
+            1);
+      }
+    } catch (Throwable e) {
+      Assert.fail("failed to produce messages", e);
+    } finally {
+      producer.shutdown();
+    }
+  }
+
+  @AfterClass
+  public void tearDown()
+      throws Exception {
+    _rocketmqCluster.close();

Review comment:
       This appears to fail in CI:
   ```
   Error:  WARNING: An illegal reflective access operation has occurred
   [24844](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24844)
   Error:  WARNING: Illegal reflective access by org.apache.rocketmq.store.MappedFile$1 (file:/home/runner/.m2/repository/org/apache/rocketmq/rocketmq-store/4.9.0/rocketmq-store-4.9.0.jar) to method java.nio.DirectByteBuffer.attachment()
   [24845](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24845)
   Error:  WARNING: Please consider reporting this to the maintainers of org.apache.rocketmq.store.MappedFile$1
   [24846](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24846)
   Error:  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
   [24847](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24847)
   Error:  WARNING: All illegal access operations will be denied in a future release
   [24848](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24848)
   Error:  Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 122.343 s <<< FAILURE! - in org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest
   [24849](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24849)
   Error:  org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.testBuildRocketMQConfig  Time elapsed: 0.105 s
   [24850](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24850)
   Error:  org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.testConsumer  Time elapsed: 97.216 s
   [24851](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24851)
   Error:  org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.testFetchOffsets  Time elapsed: 0.018 s
   [24852](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24852)
   Error:  org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.testGetPartitionCount  Time elapsed: 0.003 s
   [24853](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24853)
   Error:  org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.tearDown  Time elapsed: 2.988 s  <<< FAILURE!
   [24854](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24854)
   java.lang.IllegalStateException: java.lang.reflect.InaccessibleObjectException: Unable to make public void jdk.internal.ref.Cleaner.clean() accessible: module java.base does not "exports jdk.internal.ref" to unnamed module @42e26948
   [24855](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24855)
   	at org.apache.rocketmq.store.MappedFile$1.run(MappedFile.java:104)
   [24856](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24856)
   	at java.base/java.security.AccessController.doPrivileged(Native Method)
   [24857](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24857)
   	at org.apache.rocketmq.store.MappedFile.invoke(MappedFile.java:97)
   [24858](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24858)
   	at org.apache.rocketmq.store.MappedFile.clean(MappedFile.java:93)
   [24859](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24859)
   	at org.apache.rocketmq.store.MappedFile.cleanup(MappedFile.java:431)
   [24860](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24860)
   	at org.apache.rocketmq.store.ReferenceResource.release(ReferenceResource.java:63)
   [24861](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24861)
   	at org.apache.rocketmq.store.ReferenceResource.shutdown(ReferenceResource.java:47)
   [24862](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24862)
   	at org.apache.rocketmq.store.MappedFile.destroy(MappedFile.java:439)
   [24863](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24863)
   	at org.apache.rocketmq.store.AllocateMappedFileService.shutdown(AllocateMappedFileService.java:129)
   [24864](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24864)
   	at org.apache.rocketmq.store.DefaultMessageStore.shutdown(DefaultMessageStore.java:320)
   [24865](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24865)
   	at org.apache.rocketmq.broker.BrokerController.shutdown(BrokerController.java:770)
   [24866](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24866)
   	at org.apache.pinot.plugin.stream.rocketmq.utils.MiniRocketMQCluster.close(MiniRocketMQCluster.java:73)
   [24867](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24867)
   	at org.apache.pinot.plugin.stream.rocketmq.RocketMQStreamLevelConsumerTest.tearDown(RocketMQStreamLevelConsumerTest.java:102)
   [24868](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24868)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [24869](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24869)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [24870](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24870)
   	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [24871](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24871)
   	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [24872](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24872)
   	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
   [24873](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24873)
   	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
   [24874](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24874)
   	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
   [24875](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24875)
   	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146)
   [24876](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24876)
   	at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:212)
   [24877](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24877)
   	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
   [24878](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24878)
   	at org.testng.TestRunner.privateRun(TestRunner.java:744)
   [24879](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24879)
   	at org.testng.TestRunner.run(TestRunner.java:602)
   [24880](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24880)
   	at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
   [24881](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24881)
   	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
   [24882](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24882)
   	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
   [24883](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24883)
   	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
   [24884](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24884)
   	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
   [24885](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24885)
   	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
   [24886](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24886)
   	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
   [24887](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24887)
   	at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
   [24888](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24888)
   	at org.testng.TestNG.runSuites(TestNG.java:1144)
   [24889](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24889)
   	at org.testng.TestNG.run(TestNG.java:1115)
   [24890](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24890)
   	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136)
   [24891](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24891)
   	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
   [24892](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24892)
   	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
   [24893](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24893)
   	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145)
   [24894](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24894)
   	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
   [24895](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24895)
   	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
   [24896](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24896)
   	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
   [24897](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24897)
   	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
   [24898](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24898)
   Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make public void jdk.internal.ref.Cleaner.clean() accessible: module java.base does not "exports jdk.internal.ref" to unnamed module @42e26948
   [24899](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24899)
   	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
   [24900](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24900)
   	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
   [24901](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24901)
   	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:198)
   [24902](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24902)
   	at java.base/java.lang.reflect.Method.setAccessible(Method.java:192)
   [24903](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24903)
   	at org.apache.rocketmq.store.MappedFile$1.run(MappedFile.java:101)
   [24904](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24904)
   	... 42 more
   [24905](https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true#step:5:24905)
   ```
   
   https://github.com/apache/pinot/runs/5389825306?check_suite_focus=true




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
navina commented on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1059717586


   @0902horn Is the `partitionId` being an `int` in Pinot the only reason for implementing this connector as a `StreamLevelConsumer` ? How is it defined in RocketMQ?
   Does RocketMQ support both partition-level consumer and stream-level consumer models? 
   
   Thanks for the contribution, btw. This is great! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a change in pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
navina commented on a change in pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#discussion_r820066810



##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQStreamLevelConsumer.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamDecoderProvider;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.rocketmq.client.consumer.DefaultLitePullConsumer;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * A {@link StreamLevelConsumer} implementation for the RocketMQ stream
+ */
+public class RocketMQStreamLevelConsumer implements StreamLevelConsumer {
+
+  private StreamMessageDecoder _messageDecoder;
+  private Logger _instanceLogger;
+
+  private StreamConfig _streamConfig;
+  private RocketMQConfig _rocketmqStreamLevelStreamConfig;
+
+  private DefaultLitePullConsumer _consumer;
+  private List<MessageExt> _consumerRecords;
+  private Iterator<MessageExt> _rocketmqIterator;
+  private Map<MessageQueue, Long> _consumerOffsets = new HashMap<>(); // tracking current consumed records offsets.
+
+  private long _lastLogTime = 0;
+  private long _lastCount = 0;
+  private long _currentCount = 0L;
+
+  public RocketMQStreamLevelConsumer(String clientId, String tableName, StreamConfig streamConfig,
+      Set<String> sourceFields, String groupId) {
+    _streamConfig = streamConfig;
+    _rocketmqStreamLevelStreamConfig = new RocketMQConfig(streamConfig, groupId);
+
+    _messageDecoder = StreamDecoderProvider.create(streamConfig, sourceFields);
+
+    _instanceLogger =
+        LoggerFactory.getLogger(RocketMQConfig.class.getName() + "_" + tableName + "_" + streamConfig.getTopicName());
+    _instanceLogger.info("RocketMQStreamLevelConsumer: streamConfig : {}", _streamConfig);
+  }
+
+  @Override
+  public void start()
+      throws Exception {
+    _consumer = RocketMQStreamLevelConsumerManager.acquireRocketMQConsumerForConfig(_rocketmqStreamLevelStreamConfig);
+  }
+
+  private void updateRocketMQIterator() {
+    _consumerRecords = _consumer.poll(_streamConfig.getFetchTimeoutMillis());
+    _rocketmqIterator = _consumerRecords.iterator();
+  }
+
+  private void resetOffsets() {
+    for (MessageQueue queue : _consumerOffsets.keySet()) {
+      long offsetToSeek = _consumerOffsets.get(queue);
+      try {
+        _consumer.seek(queue, offsetToSeek);
+      } catch (MQClientException e) {
+        _instanceLogger.warn("Caught exception while seek offset", e);
+      }
+    }
+  }
+
+  @Override
+  public GenericRow next(GenericRow destination) {
+    if (_rocketmqIterator == null || !_rocketmqIterator.hasNext()) {
+      updateRocketMQIterator();
+    }
+    if (_rocketmqIterator.hasNext()) {
+      try {
+        final MessageExt record = _rocketmqIterator.next();
+        updateOffsets(new MessageQueue(record.getTopic(), record.getBrokerName(), record.getQueueId()),
+            record.getQueueOffset());
+        destination = _messageDecoder.decode(record.getBody(), destination);
+
+        _currentCount++;
+
+        final long now = System.currentTimeMillis();
+        // Log every minute or 100k events
+        if (now - _lastLogTime > 60000 || _currentCount - _lastCount >= 100000) {
+          if (_lastCount == 0) {
+            _instanceLogger
+                .info("Consumed {} events from rocketmq stream {}", _currentCount, _streamConfig.getTopicName());
+          } else {
+            _instanceLogger.info("Consumed {} events from rocketmq stream {} (rate:{}/s)", _currentCount - _lastCount,
+                _streamConfig.getTopicName(), (float) (_currentCount - _lastCount) * 1000 / (now - _lastLogTime));
+          }
+          _lastCount = _currentCount;
+          _lastLogTime = now;
+        }
+        return destination;
+      } catch (Exception e) {
+        _instanceLogger.warn("Caught exception while consuming events", e);
+        throw e;
+      }
+    }
+    return null;
+  }
+
+  private void updateOffsets(MessageQueue queue, long offset) {

Review comment:
       I have some questions about `MessageQueue`:
   1. Does the consumer maintain a queue per broker per topic ? 
   2. In RocketMQ, is the message ordering guaranteed for messages within the same broker? 
   

##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQConsumerFactory.java
##########
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.Set;
+import org.apache.pinot.spi.stream.PartitionLevelConsumer;
+import org.apache.pinot.spi.stream.StreamConsumerFactory;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMetadataProvider;
+
+
+/**
+ * A {@link StreamConsumerFactory} implementation for the RocketMQ stream
+ */
+public class RocketMQConsumerFactory extends StreamConsumerFactory {
+  @Override
+  public PartitionLevelConsumer createPartitionLevelConsumer(String clientId, int partition) {
+    throw new UnsupportedOperationException();

Review comment:
       nit: It will be useful if you can include a message with the exception saying "RocketMQ doesn't support partition level consumer".
   Same comment for Line 45 as well. 

##########
File path: pinot-plugins/pinot-stream-ingestion/pinot-rocketmq/src/main/java/org/apache/pinot/plugin/stream/rocketmq/RocketMQStreamLevelConsumer.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.stream.rocketmq;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.stream.StreamConfig;
+import org.apache.pinot.spi.stream.StreamDecoderProvider;
+import org.apache.pinot.spi.stream.StreamLevelConsumer;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.rocketmq.client.consumer.DefaultLitePullConsumer;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * A {@link StreamLevelConsumer} implementation for the RocketMQ stream
+ */
+public class RocketMQStreamLevelConsumer implements StreamLevelConsumer {
+
+  private StreamMessageDecoder _messageDecoder;
+  private Logger _instanceLogger;
+
+  private StreamConfig _streamConfig;
+  private RocketMQConfig _rocketmqStreamLevelStreamConfig;
+
+  private DefaultLitePullConsumer _consumer;
+  private List<MessageExt> _consumerRecords;
+  private Iterator<MessageExt> _rocketmqIterator;
+  private Map<MessageQueue, Long> _consumerOffsets = new HashMap<>(); // tracking current consumed records offsets.
+
+  private long _lastLogTime = 0;
+  private long _lastCount = 0;
+  private long _currentCount = 0L;
+
+  public RocketMQStreamLevelConsumer(String clientId, String tableName, StreamConfig streamConfig,
+      Set<String> sourceFields, String groupId) {
+    _streamConfig = streamConfig;
+    _rocketmqStreamLevelStreamConfig = new RocketMQConfig(streamConfig, groupId);
+
+    _messageDecoder = StreamDecoderProvider.create(streamConfig, sourceFields);
+
+    _instanceLogger =
+        LoggerFactory.getLogger(RocketMQConfig.class.getName() + "_" + tableName + "_" + streamConfig.getTopicName());
+    _instanceLogger.info("RocketMQStreamLevelConsumer: streamConfig : {}", _streamConfig);
+  }
+
+  @Override
+  public void start()
+      throws Exception {
+    _consumer = RocketMQStreamLevelConsumerManager.acquireRocketMQConsumerForConfig(_rocketmqStreamLevelStreamConfig);
+  }
+
+  private void updateRocketMQIterator() {
+    _consumerRecords = _consumer.poll(_streamConfig.getFetchTimeoutMillis());
+    _rocketmqIterator = _consumerRecords.iterator();
+  }
+
+  private void resetOffsets() {
+    for (MessageQueue queue : _consumerOffsets.keySet()) {
+      long offsetToSeek = _consumerOffsets.get(queue);
+      try {
+        _consumer.seek(queue, offsetToSeek);
+      } catch (MQClientException e) {
+        _instanceLogger.warn("Caught exception while seek offset", e);
+      }
+    }
+  }
+
+  @Override
+  public GenericRow next(GenericRow destination) {
+    if (_rocketmqIterator == null || !_rocketmqIterator.hasNext()) {
+      updateRocketMQIterator();
+    }
+    if (_rocketmqIterator.hasNext()) {
+      try {
+        final MessageExt record = _rocketmqIterator.next();
+        updateOffsets(new MessageQueue(record.getTopic(), record.getBrokerName(), record.getQueueId()),
+            record.getQueueOffset());
+        destination = _messageDecoder.decode(record.getBody(), destination);
+
+        _currentCount++;
+
+        final long now = System.currentTimeMillis();
+        // Log every minute or 100k events
+        if (now - _lastLogTime > 60000 || _currentCount - _lastCount >= 100000) {
+          if (_lastCount == 0) {
+            _instanceLogger
+                .info("Consumed {} events from rocketmq stream {}", _currentCount, _streamConfig.getTopicName());
+          } else {
+            _instanceLogger.info("Consumed {} events from rocketmq stream {} (rate:{}/s)", _currentCount - _lastCount,
+                _streamConfig.getTopicName(), (float) (_currentCount - _lastCount) * 1000 / (now - _lastLogTime));
+          }
+          _lastCount = _currentCount;
+          _lastLogTime = now;
+        }
+        return destination;
+      } catch (Exception e) {
+        _instanceLogger.warn("Caught exception while consuming events", e);
+        throw e;
+      }
+    }
+    return null;
+  }
+
+  private void updateOffsets(MessageQueue queue, long offset) {
+    _consumerOffsets.put(queue, offset + 1);

Review comment:
       Also, do we need to track the current consumed offset? It doesn't look like it is being "read" anywhere? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 0902horn edited a comment on pull request #8274: Rocketmq integration

Posted by GitBox <gi...@apache.org>.
0902horn edited a comment on pull request #8274:
URL: https://github.com/apache/pinot/pull/8274#issuecomment-1060163126


   > Is the `partitionId` being an `int` in Pinot the only reason for implementing this connector as a `StreamLevelConsumer` ? 
   
   Yes. 
   
   > How is it defined in RocketMQ?
   
   MessageQueue is defined as a  3-tuple like *(topic, broker, queueId)*. Notice that *queueId* is a number in the broker scoped.
   
   > Does RocketMQ support both partition-level consumer and stream-level consumer models?
   
   Yes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org