You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "glcrazier (via GitHub)" <gi...@apache.org> on 2024/06/09 12:51:12 UTC

[PR] [rust] implement standard pushconsumer [rocketmq-clients]

glcrazier opened a new pull request, #767:
URL: https://github.com/apache/rocketmq-clients/pull/767

   <!-- Please make sure the target branch is right. In most case, the target branch should be `master`. -->
   
   ### Which Issue(s) This PR Fixes
   
   <!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
   
   Fixes #issue_id
   
   ### Brief Description
   
   <!-- Write a brief description for your pull request to help the maintainer understand the reasons behind your changes. -->
   This is an implementation of standard PushConsumer which can consumes messages with listener provided. 
   
   ### How Did You Test This Change?
   
   <!-- In order to ensure the code quality of Apache RocketMQ, we expect every pull request to have undergone thorough testing. -->
   Sufficient unit test cases are provided and integration tests are executed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#issuecomment-2244109529

   ## [Codecov](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `85.61976%` with `326 lines` in your changes missing coverage. Please review.
   > Project coverage is 52.48%. Comparing base [(`0149785`)](https://app.codecov.io/gh/apache/rocketmq-clients/commit/0149785a7d11ed9d8bccca3c2049283bfedd4e7c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`9ee7f92`)](https://app.codecov.io/gh/apache/rocketmq-clients/commit/9ee7f920563619ff8192a58d340a627f23517ebd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 13 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [rust/src/push\_consumer.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fpush_consumer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvcHVzaF9jb25zdW1lci5ycw==) | 86.99% | [128 Missing and 62 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/client.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fclient.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvY2xpZW50LnJz) | 72.66% | [68 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/session.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fsession.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvc2Vzc2lvbi5ycw==) | 71.73% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/conf.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fconf.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvY29uZi5ycw==) | 92.43% | [14 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/producer.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fproducer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvcHJvZHVjZXIucnM=) | 81.25% | [2 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/util.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Futil.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvdXRpbC5ycw==) | 97.14% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/model/message.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fmodel%2Fmessage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvbW9kZWwvbWVzc2FnZS5ycw==) | 71.42% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/model/common.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fmodel%2Fcommon.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvbW9kZWwvY29tbW9uLnJz) | 97.22% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [rust/src/simple\_consumer.rs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&filepath=rust%2Fsrc%2Fsimple_consumer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVzdC9zcmMvc2ltcGxlX2NvbnN1bWVyLnJz) | 85.71% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master     #767      +/-   ##
   ============================================
   - Coverage     56.01%   52.48%   -3.53%     
     Complexity      651      651              
   ============================================
     Files           259      208      -51     
     Lines         15836    14243    -1593     
     Branches       4069     5595    +1526     
   ============================================
   - Hits           8871     7476    -1395     
   + Misses         6676     6421     -255     
   - Partials        289      346      +57     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [nodejs](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [rust](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.04% <85.61%> (+2.07%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq-clients/pull/767?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673306010


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;

Review Comment:
   Fn requires immutability, kind of excessively strict. Use FnMut?
   Here is a good reference to read:
   https://stackoverflow.com/questions/41081240/idiomatic-callbacks-in-rust



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "glcrazier (via GitHub)" <gi...@apache.org>.
glcrazier commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673495957


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;

Review Comment:
   The FnMut requires exclusive access in threads which means incapable of concurrent consumption. Do you have any idea?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673333147


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);
+                match consume_result {
+                    ConsumeResult::SUCCESS => {
+                        let result = self.ack_message(rpc_client, &message).await;
+                        if result.is_err() {
+                            info!(

Review Comment:
   It would be better to define an abstract model here, some kind of ProcessTask and encapsulate process-ack-retry and order-keeping.
   That is, an message queue actor has one(FIFO) or multiple(standard model) ProcessTask, one message arrival, put messages to existing ProcessTask(for FIFO) or spawn new ProcessTasks.
   
   In the process task, we have the consume semantics implemented.
   
   This way, readability shall be improved and code would be sustainable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1705339663


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;

Review Comment:
   Let's merge it ASAP and file a new pull request to improve.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673330758


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);
+                match consume_result {
+                    ConsumeResult::SUCCESS => {
+                        let result = self.ack_message(rpc_client, &message).await;
+                        if result.is_err() {

Review Comment:
   Ah, you have implemented the retry mechanism.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673333147


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);
+                match consume_result {
+                    ConsumeResult::SUCCESS => {
+                        let result = self.ack_message(rpc_client, &message).await;
+                        if result.is_err() {
+                            info!(

Review Comment:
   It would be better to define an abstract model here, some kind of ProcessTask and encapsulate process-ack-retry and order-keeping.
   That is, an message queue actor has one(FIFO) or multiple(standard model) ProcessTask, one message arrival, put messages to existing ProcessTask(for FIFO) or spawn new ProcessTasks.
   
   In the process task, we have the consume semantics implemented.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "ShadowySpirits (via GitHub)" <gi...@apache.org>.
ShadowySpirits commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1705444520


##########
.github/workflows/rust_build.yml:
##########
@@ -45,27 +52,26 @@ jobs:
         with:
           toolchain: stable
       - name: Build doc
-        working-directory: ./rust
         run: cargo doc --no-deps --all-features
         env:
           RUSTDOCFLAGS: --cfg docsrs
   msrv:
     runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ./rust
     strategy:
       matrix:
-        msrv: [1.61]

Review Comment:
   Rust version defined in Cargo.toml should be also bumped up to 1.70.0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "glcrazier (via GitHub)" <gi...@apache.org>.
glcrazier commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1685358156


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);
+                match consume_result {
+                    ConsumeResult::SUCCESS => {
+                        let result = self.ack_message(rpc_client, &message).await;
+                        if result.is_err() {
+                            info!(

Review Comment:
   You idea is great, and I refactored that the consumption of standard and fifo is capsulated in separate implementation. And I also extract the acknowledgement work into a single implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "glcrazier (via GitHub)" <gi...@apache.org>.
glcrazier commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1685358315


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);

Review Comment:
   Now it's done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui merged PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#issuecomment-2271053671

   MSRV set to 1.61 does not make sense 
   run cargo msrv, we are having 
   <img width="1061" alt="image" src="https://github.com/user-attachments/assets/483a539c-ebfd-4018-92b0-d2df88af9b15">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#issuecomment-2165648870

   Thanks @glcrazier for your nice work and I would join you and review this promising feature ASAP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673324426


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);

Review Comment:
   On message arrival, we will consume the message in the current task? This would limit the parallelism of message processing to # of actors.
   
   For concurrent model, in which out-of-order execution is allowed, we shall spawn a task for each mini-batch of messages, size of the mini-batch is up to configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673324842


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);

Review Comment:
   For FIFO model, this is exactly what we need and we shall keep it as it is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] implement standard pushconsumer [rocketmq-clients]

Posted by "lizhanhui (via GitHub)" <gi...@apache.org>.
lizhanhui commented on code in PR #767:
URL: https://github.com/apache/rocketmq-clients/pull/767#discussion_r1673327287


##########
rust/src/push_consumer.rs:
##########
@@ -0,0 +1,1303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+use mockall::automock;
+use mockall_double::double;
+use parking_lot::{Mutex, RwLock};
+use prost_types::Duration;
+use slog::Logger;
+use slog::{debug, error, info, warn};
+use std::collections::{HashMap, VecDeque};
+use std::sync::Arc;
+use tokio::select;
+use tokio::sync::{mpsc, oneshot};
+
+#[double]
+use crate::client::Client;
+use crate::conf::{BackOffRetryPolicy, ClientOption, PushConsumerOption};
+use crate::error::{ClientError, ErrorKind};
+use crate::model::common::{ClientType, ConsumeResult, FilterExpression, MessageQueue};
+use crate::model::message::{AckMessageEntry, MessageView};
+use crate::pb::receive_message_response::Content;
+use crate::pb::{
+    AckMessageRequest, Assignment, ChangeInvisibleDurationRequest, QueryAssignmentRequest,
+    ReceiveMessageRequest, Resource,
+};
+use crate::session::{RPCClient, Session};
+use crate::util::{build_endpoints_by_message_queue, build_push_consumer_settings};
+use crate::{log, pb};
+
+const OPERATION_NEW_PUSH_CONSUMER: &str = "push_consumer.new";
+const OPERATION_RECEIVE_MESSAGE: &str = "push_consumer.receive_message";
+const OPERATION_ACK_MESSAGE: &str = "push_consumer.ack_message";
+const OPERATION_START_PUSH_CONSUMER: &str = "push_consumer.start";
+const OPERATION_CHANGE_INVISIBLE_DURATION: &str = "push_consumer.change_invisible_duration";
+
+pub type MessageListener = Box<dyn Fn(&MessageView) -> ConsumeResult + Send + Sync>;
+
+pub struct PushConsumer {
+    logger: Logger,
+    client: Client,
+    message_listener: Arc<MessageListener>,
+    option: Arc<RwLock<PushConsumerOption>>,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+}
+
+/*
+ * An actor is required for each message queue.
+ * It is responsible for polling messages from the message queue. It communicates with PushConsumer by channels.
+ */
+struct MessageQueueActor {
+    logger: Logger,
+    rpc_client: Session,
+    message_queue: MessageQueue,
+    shutdown_tx: Option<oneshot::Sender<()>>,
+    option: PushConsumerOption,
+    message_listener: Arc<MessageListener>,
+    retry_policy: BackOffRetryPolicy,
+}
+
+impl PushConsumer {
+    pub fn new(
+        client_option: ClientOption,
+        option: PushConsumerOption,
+        message_listener: MessageListener,
+    ) -> Result<Self, ClientError> {
+        if option.consumer_group().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "consumer group is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        if option.subscription_expressions().is_empty() {
+            return Err(ClientError::new(
+                ErrorKind::Config,
+                "subscription expressions is required.",
+                OPERATION_NEW_PUSH_CONSUMER,
+            ));
+        }
+        let client_option = ClientOption {
+            client_type: ClientType::PushConsumer,
+            group: Some(option.consumer_group().to_string()),
+            ..client_option
+        };
+        let logger = log::logger(option.logging_format());
+        let client = Client::new(
+            &logger,
+            client_option,
+            build_push_consumer_settings(&option),
+        )?;
+        Ok(Self {
+            logger,
+            client,
+            message_listener: Arc::new(message_listener),
+            option: Arc::new(RwLock::new(option)),
+            shutdown_tx: None,
+        })
+    }
+
+    pub async fn start(&mut self) -> Result<(), ClientError> {
+        let (telemetry_command_tx, mut telemetry_command_rx) = mpsc::channel(16);
+        self.client.start(telemetry_command_tx).await?;
+        let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(shutdown_tx);
+        let option = Arc::clone(&self.option);
+        let mut rpc_client = self.client.get_session().await?;
+        let route_manager = self.client.get_route_manager();
+        let topics;
+        {
+            topics = option
+                .read()
+                .subscription_expressions()
+                .keys()
+                .cloned()
+                .collect();
+        }
+        route_manager
+            .sync_topic_routes(&mut rpc_client, topics)
+            .await?;
+        let logger = self.logger.clone();
+        let mut actor_table: HashMap<MessageQueue, MessageQueueActor> = HashMap::new();
+
+        let message_listener = Arc::clone(&self.message_listener);
+        // must retrieve settings from server first.
+        let command = telemetry_command_rx.recv().await.ok_or(ClientError::new(
+            ErrorKind::Connect,
+            "telemetry command channel closed.",
+            OPERATION_START_PUSH_CONSUMER,
+        ))?;
+        let backoff_policy = Arc::new(Mutex::new(BackOffRetryPolicy::try_from(command).map_err(
+            |_| {
+                ClientError::new(
+                    ErrorKind::Connect,
+                    "backoff policy is required.",
+                    OPERATION_START_PUSH_CONSUMER,
+                )
+            },
+        )?));
+
+        tokio::spawn(async move {
+            let mut scan_assignments_timer =
+                tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                select! {
+                    command = telemetry_command_rx.recv() => {
+                        if let Some(command) = command {
+                            let remote_backoff_policy = BackOffRetryPolicy::try_from(command);
+                            if let Ok(remote_backoff_policy) = remote_backoff_policy {
+                                let mut guard = backoff_policy.lock();
+                                *guard = remote_backoff_policy;
+                            }
+                        }
+                    }
+                    _ = scan_assignments_timer.tick() => {
+                        let consumer_option;
+                        {
+                            consumer_option = option.read().clone();
+                        }
+                        let subscription_table = consumer_option.subscription_expressions();
+                        // query endpoints from topic route
+                        let retry_policy = Arc::clone(&backoff_policy);
+                        let mut client = rpc_client.shadow_session();
+                        for topic in subscription_table.keys() {
+                            if let Some(endpoints) = route_manager.pick_endpoints(topic.as_str()) {
+                                let request = QueryAssignmentRequest {
+                                    topic: Some(Resource {
+                                        name: topic.to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    group: Some(Resource {
+                                        name: consumer_option.consumer_group().to_string(),
+                                        resource_namespace: consumer_option.namespace().to_string(),
+                                    }),
+                                    endpoints: Some(endpoints.into_inner()),
+                                };
+                                let retry_policy_inner;
+                                {
+                                    retry_policy_inner = retry_policy.lock().clone();
+                                }
+                                let result = client.query_assignment(request).await;
+                                if let Ok(response) = result {
+                                    if Client::handle_response_status(response.status, OPERATION_START_PUSH_CONSUMER).is_ok() {
+                                            let _ = Self::process_assignments(logger.clone(),
+                                                rpc_client.shadow_session(),
+                                                &consumer_option,
+                                                Arc::clone(&message_listener),
+                                                &mut actor_table,
+                                                response.assignments,
+                                                retry_policy_inner,
+                                            ).await;
+                                    } else {
+                                        error!(logger, "query assignment failed, no status in response.");
+                                    }
+                                } else {
+                                    error!(logger, "query assignment failed: {:?}", result.unwrap_err());
+                                }
+                            }
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        let entries = actor_table.drain();
+                        info!(logger, "shutdown {:?} actors", entries.len());
+                        for (_, actor) in entries {
+                            let _ = actor.shutdown().await;
+                        }
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn process_assignments(
+        logger: Logger,
+        rpc_client: Session,
+        option: &PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        actor_table: &mut HashMap<MessageQueue, MessageQueueActor>,
+        mut assignments: Vec<Assignment>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Result<(), ClientError> {
+        let message_queues: Vec<MessageQueue> = assignments
+            .iter_mut()
+            .filter_map(|assignment| {
+                if let Some(message_queue) = assignment.message_queue.take() {
+                    return MessageQueue::from_pb_message_queue(message_queue).ok();
+                }
+                None
+            })
+            .collect();
+        // remove existing actors from map, the remaining ones will be shutdown.
+        let mut actors: Vec<MessageQueueActor> = Vec::with_capacity(actor_table.len());
+        message_queues.iter().for_each(|message_queue| {
+            let entry = actor_table.remove(message_queue);
+            if let Some(actor) = entry {
+                actors.push(actor);
+            }
+        });
+        // the remaining actors will be shutdown.
+        let shutdown_entries = actor_table.drain();
+        for (_, actor) in shutdown_entries {
+            let _ = actor.shutdown().await;
+        }
+        let mut max_cache_messages_per_queue = option.max_cache_message_count();
+        if !message_queues.is_empty() {
+            max_cache_messages_per_queue = option.max_cache_message_count() / message_queues.len();
+        }
+        for mut actor in actors {
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            actor.set_option(option);
+            actor.set_retry_policy(retry_policy.clone());
+            actor_table.insert(actor.message_queue.clone(), actor);
+        }
+
+        // start new actors
+        for message_queue in message_queues {
+            if actor_table.contains_key(&message_queue) {
+                continue;
+            }
+            let mut option = option.clone();
+            option.set_max_cache_message_count(max_cache_messages_per_queue);
+            let mut actor = MessageQueueActor::new(
+                logger.clone(),
+                rpc_client.shadow_session(),
+                message_queue.clone(),
+                option,
+                Arc::clone(&message_listener),
+                retry_policy.clone(),
+            );
+            let result = actor.start().await;
+            if result.is_ok() {
+                actor_table.insert(message_queue, actor);
+            }
+        }
+        Ok(())
+    }
+
+    pub async fn shutdown(mut self) -> Result<(), ClientError> {
+        if let Some(shutdown_tx) = self.shutdown_tx.take() {
+            let _ = shutdown_tx.send(());
+        }
+        self.client.shutdown().await?;
+        Ok(())
+    }
+}
+
+struct MessageHandler {
+    message: MessageView,
+    status: ConsumeResult,
+    attempt: usize,
+}
+
+#[automock]
+impl MessageQueueActor {
+    pub(crate) fn new(
+        logger: Logger,
+        rpc_client: Session,
+        message_queue: MessageQueue,
+        option: PushConsumerOption,
+        message_listener: Arc<MessageListener>,
+        retry_policy: BackOffRetryPolicy,
+    ) -> Self {
+        Self {
+            logger,
+            rpc_client,
+            message_queue,
+            shutdown_tx: None,
+            option,
+            message_listener: Arc::clone(&message_listener),
+            retry_policy,
+        }
+    }
+
+    // The following two methods won't affect the running actor in fact.
+    pub(crate) fn set_option(&mut self, option: PushConsumerOption) {
+        self.option = option;
+    }
+
+    pub(crate) fn set_retry_policy(&mut self, retry_policy: BackOffRetryPolicy) {
+        self.retry_policy = retry_policy;
+    }
+
+    fn fork_self(&self) -> Self {
+        Self {
+            logger: self.logger.clone(),
+            rpc_client: self.rpc_client.shadow_session(),
+            message_queue: self.message_queue.clone(),
+            shutdown_tx: None,
+            option: self.option.clone(),
+            message_listener: Arc::clone(&self.message_listener),
+            retry_policy: self.retry_policy.clone(),
+        }
+    }
+
+    pub(crate) fn get_consumer_group(&self) -> Resource {
+        Resource {
+            name: self.option.consumer_group().to_string(),
+            resource_namespace: self.option.namespace().to_string(),
+        }
+    }
+
+    #[allow(clippy::needless_lifetimes)]
+    pub(crate) fn get_filter_expression<'a>(&'a self) -> Option<&'a FilterExpression> {
+        self.option
+            .subscription_expressions()
+            .get(self.message_queue.topic.name.as_str())
+    }
+
+    pub(crate) async fn start(&mut self) -> Result<(), ClientError> {
+        debug!(
+            self.logger,
+            "start a new queue actor {:?}", self.message_queue
+        );
+        let (tx, mut shutdown_rx) = oneshot::channel();
+        self.shutdown_tx = Some(tx);
+        let (tx, mut poll_rx) = mpsc::channel(8);
+        let poll_tx: mpsc::Sender<()> = tx;
+        let logger = self.logger.clone();
+        let mut actor = self.fork_self();
+        let mut message_handler_queue: VecDeque<MessageHandler> = VecDeque::new();
+        let (tx, mut ack_rx) = mpsc::channel(8);
+        let ack_tx: mpsc::Sender<()> = tx;
+        let mut queue_check_ticker = tokio::time::interval(std::time::Duration::from_secs(1));
+        let mut rpc_client = actor.rpc_client.shadow_session();
+        tokio::spawn(async move {
+            loop {
+                select! {
+                    _ = poll_rx.recv() => {
+                        actor.receive_messages(&mut rpc_client, &mut message_handler_queue, poll_tx.clone()).await;
+                    }
+                    _ = ack_rx.recv() => {
+                        actor.ack_message_in_waiting_queue(&mut rpc_client, &mut message_handler_queue, ack_tx.clone()).await;
+                    }
+                    _ = queue_check_ticker.tick() => {
+                        if !message_handler_queue.is_empty() {
+                            let _ = ack_tx.try_send(());
+                        }
+                        if message_handler_queue.len() < actor.option.max_cache_message_count() {
+                            let _ = poll_tx.try_send(());
+                        }
+                    }
+                    _ = &mut shutdown_rx => {
+                        info!(logger, "message queue actor shutdown");
+                        actor.flush_waiting_queue(&mut rpc_client, &mut message_handler_queue).await;
+                        break;
+                    }
+                }
+            }
+        });
+        Ok(())
+    }
+
+    async fn receive_messages<T: RPCClient + 'static>(
+        &mut self,
+        rpc_client: &mut T,
+        message_handler_queue: &mut VecDeque<MessageHandler>,
+        poll_tx: mpsc::Sender<()>,
+    ) {
+        let poll_result = self.poll_messages(rpc_client).await;
+        if let Ok(messages) = poll_result {
+            for message in messages {
+                let consume_result = (self.message_listener)(&message);
+                match consume_result {
+                    ConsumeResult::SUCCESS => {
+                        let result = self.ack_message(rpc_client, &message).await;
+                        if result.is_err() {

Review Comment:
   Error handling is not sufficient here.
   For FIFO messages, we have to repeatedly retry ack the message before getting opportunity  to consume the next one.
   
   For standard message model, we also need to retry because consume a message might be expensive in terms of resource consumption.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org