You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "hhuang1231 (via GitHub)" <gi...@apache.org> on 2023/12/04 01:01:52 UTC
[PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
hhuang1231 opened a new pull request, #4599:
URL: https://github.com/apache/eventmesh/pull/4599
Fixes #4598 .
### Motivation
Improve eventmesh-connector-feishu.
### Modifications
See #4598 for details.
### Output

### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838027901
> I have question about why you change the module name from Feishu to lark ?
I found that the dingding module was changed to dingtalk.So I think it is necessary to change Feishu to lark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838286037
@Pil0tXia Yes, I registered accounts for lark and feishu, and after testing, they can sink events.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413399357
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
If it is possible to develop other services that require this credential to further enrich the connector, you can leave a note in this method for other pending services that may require this credential.
如果还有可能开发其他需要该凭证的服务来进一步丰富该Connector,您可以在该方法备注一下可能需要该凭证的其他等待实现的服务。
> Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?
I encountered similar doubts in an other connector before https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091, and the developer explained that because credentials will expire and new credential will be constantly obtained in the code https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666. I'm not sure if the credential here will expire. If it do, not only does it require Cache, but also a mechanism for regular updates.
我之前在其他Connector遇到类似的疑惑 https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091 ,开发者解释到因为凭证会过期 https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666 ,并且在代码中会不断获取新的凭证。我不清楚这里的凭证会不会过期,如果会过期,不仅需要Cache,还需要定时更新的机制。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413516292
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
Lark's access credentials have an expiration mechanism, but the expiration time can be refreshed by re-obtaining new credentials within half an hour before expiration. The documentation provided by Lark is as follows:
> The maximum validity period of tenant_access_token is 2 hours. If this interface is called when the validity period is less than 30 minutes, a new tenant_access_token will be returned, and there will be two valid tenant_access_tokens at the same time.
_I think expiration is not the reason for using Cache. If you only need to manage one kind of credential, then there is no need to use a collection to maintain the validity of the credential._
---
飞书的访问凭证存在过期机制,但是在过期前半个小时内重新获取新的凭证即可刷新过期时间。飞书提供的文档说明如下:
> tenant_access_token 的最大有效期是 2 小时。如果在有效期小于 30 分钟的情况下,调用本接口,会返回一个新的 tenant_access_token,这会同时存在两个有效的 tenant_access_token。
_我认为过期并不是使用Cache的原因,如果仅需要管理一种凭证,那么并不需要使用一个集合来维护凭证的有效性。_
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
Lark's access credentials have an expiration mechanism, but the expiration time can be refreshed by re-obtaining new credentials within half an hour before expiration. The documentation provided by Lark is as follows:
> The maximum validity period of tenant_access_token is 2 hours. If this interface is called when the validity period is less than 30 minutes, a new tenant_access_token will be returned, and there will be two valid tenant_access_tokens at the same time.
_I think expiration is not the reason for using Cache. If you only need to manage one kind of credential, then there is no need to use a collection to maintain the validity of the credential._
---
飞书的访问凭证存在过期机制,但是在过期前半个小时内重新获取新的凭证即可刷新过期时间。飞书提供的文档说明如下:
> tenant_access_token 的最大有效期是 2 小时。如果在有效期小于 30 分钟的情况下,调用本接口,会返回一个新的 tenant_access_token,这会同时存在两个有效的 tenant_access_token。
_我认为过期并不是使用Cache的原因,如果仅需要管理一种凭证,那么并不需要使用一个集合来维护凭证的有效性。_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415877327
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
ok,i will try to refactor it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415752257
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
+ } else {
+ Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413635990
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
ok, this is a bug. I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413372603
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
In addition to configuration reading problems, the original Feishu Sink also had CouldEvent.Data reading problems:
原先的Feishu Sink除了配置读取问题,还有CouldEvent.Data的读取问题:

The following message will probably be generated:
大概会生成如下的消息:

##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
In addition to configuration reading problems, the original Feishu Sink also had CouldEvent.Data reading problems:
原先的Feishu Sink除了配置读取问题,还有CouldEvent.Data的读取问题:

The following message will probably be generated:
大概会生成如下的消息:

##########
eventmesh-connectors/eventmesh-connector-lark/build.gradle:
##########
@@ -15,19 +15,20 @@
* limitations under the License.
*/
-List feishu = [
- "com.larksuite.oapi:oapi-sdk:$feishu_version",
+List lark = [
+ "com.larksuite.oapi:oapi-sdk:$lark_version",
"com.github.rholder:guava-retrying:$guava_retrying_version",
"org.apache.httpcomponents:httpclient",
project(":eventmesh-common")
]
dependencies {
api project(":eventmesh-openconnect:eventmesh-openconnect-java")
- implementation feishu
+ implementation lark
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
- testImplementation "org.mockito:mockito-core"
+ //testImplementation "org.mockito:mockito-core"
Review Comment:
I will remove it.
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
+
+ private SinkConnectorConfig sinkConnectorConfig;
+
+ private ImService imService;
+
+ private RequestOptions requestOptions;
+
+ private Retryer<ConnectRecord> retryer;
+
+ public ImServiceHandler() {}
+
+ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
+ ImServiceHandler imServiceHandler = new ImServiceHandler();
+ imServiceHandler.sinkConnectorConfig = sinkConnectorConfig;
+ imServiceHandler.imService = Client.newBuilder(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret())
+ .httpTransport(new OkHttpTransport(new OkHttpClient().newBuilder()
+ .callTimeout(3L, TimeUnit.SECONDS)
+ .build())
+ )
+ .disableTokenCache()
+ .requestTimeout(3, TimeUnit.SECONDS)
+ .build()
+ .im();
+
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
+
+ imServiceHandler.requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
+ if (times == maxRetryTimes) {
+ UN_ACK_REQ.remove(attempt.get());
+ }
+ }
+ })
+ .build();
+ return imServiceHandler;
+ }
+
+ public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
+ CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
+ .receiveId(sinkConnectorConfig.getReceiveId())
+ .uuid(UUID.randomUUID().toString());
+
+ String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
+ if (null == templateTypeKey || "null".equals(templateTypeKey)) {
+ templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
+ }
+ LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
+ if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
+ bodyBuilder.content(createTextContent(connectRecord))
+ .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
+ } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
+ String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
+ .orElse("EventMesh-Message");
+ bodyBuilder.content(createInteractiveContent(connectRecord, title))
+ .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
+ }
+
+ return CreateMessageReq.newBuilder()
+ .receiveIdType(sinkConnectorConfig.getReceiveIdType())
+ .createMessageReqBody(bodyBuilder.build())
+ .build();
+ });
+ CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
+ if (resp.getCode() != 0) {
+ log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
+ return connectRecord;
+ }
+
+ UN_ACK_REQ.remove(connectRecord);
+ return null;
+ });
+ }
+
+ private String createTextContent(ConnectRecord connectRecord) {
+ MessageText.Builder msgBuilder = MessageText.newBuilder();
+
+ if (needAtAll(connectRecord)) {
+ msgBuilder.atAll();
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ msgBuilder.atUser(kv[0], kv[1]);
+ }
+ }
+
+ String escapedString = StringEscapeUtils.escapeJava(new String((byte[]) connectRecord.getData()));
+ return msgBuilder.text(escapedString).build();
+ }
+
+ private String createInteractiveContent(ConnectRecord connectRecord, String title) {
+ StringBuilder sb = new StringBuilder();
+ if (needAtAll(connectRecord)) {
+ atAll(sb);
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ atUser(sb, kv[0]);
+ }
+ }
+ sb.append(new String((byte[]) connectRecord.getData()));
+
+ MessageCardConfig config = MessageCardConfig.newBuilder()
+ .enableForward(true)
+ .wideScreenMode(true)
+ .updateMulti(true)
+ .build();
+
+ // cardUrl, Only in need of PC, mobile side jump different links to use
+ /*MessageCardURL cardURL = MessageCardURL.newBuilder()
+ .pcUrl("http://www.baidu.com")
+ .iosUrl("http://www.google.com")
+ .url("http://open.feishu.com")
+ .androidUrl("http://www.jianshu.com")
+ .build();*/
Review Comment:
I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413467974
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/ConnectRecordExtensionKeys.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark;
+
+/**
+ * Constants of record extension key.
+ */
+public interface ConnectRecordExtensionKeys {
+
+ /**
+ * {@code text} or {@code markdown}, otherwise use {@code text} to replace.
+ */
+ String TEMPLATE_TYPE_4_LARK = "templatetype4lark";
+
+ /**
+ * The value format is {@code id,name;id,name;}. <br/>
Review Comment:
I will replace them by `<p>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413500689
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
@pandaapo
If it is an asynchronous request, Feishu may fail to respond in time and the connectorServer starts to retransmit. This may lead to repeated consumption on Feishu.
But there is currently a problem with my implementation logic, because the method of using `com.github.rholder.retry.Retryer` is a synchronous request, and I mistakenly thought it was an asynchronous request.
- For synchronous requests, this collection is not needed to store unresponsive requests.
- Or think about it from another angle, where can dead letter events be stored?
my thoughts:
1. If there are performance requirements, I recommend removing `retryer` and using `ScheduledExecutorService` to implement asynchronous requests.
2. Continue to use `retryer` and remove the `UN_ACK_REQ` collection
3. If there is no need to retransmit, you can directly remove this function.
Do you have any better suggestions in this regard?
---
如果是异步请求就可能会出现飞书未能及时响应,而connectorServer开始重传的情况。这可能会导致飞书重复消费。
但是目前我的实现逻辑存在问题,因为使用`com.github.rholder.retry.Retryer`的方式属于同步请求,而我误以为是异步请求。
- 对于同步请求来说,并不需要这种集合来存储未响应的请求。
- 或者可以换另一个角度思考,这里可以存储死信事件?
我的想法:
1. 有性能需求的话,我建议移除`retryer`,自行使用`ScheduledExecutorService`实现异步请求。
2. 继续使用`retryer`,移除`UN_ACK_REQ`集合
3. 没必要重传的话,可以直接移除该功能。
请问您在这方面有什么更好的建议吗?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
@pandaapo
If it is an asynchronous request, Feishu may fail to respond in time and the connectorServer starts to retransmit. This may lead to repeated consumption on Feishu.
But there is currently a problem with my implementation logic, because the method of using `com.github.rholder.retry.Retryer` is a synchronous request, and I mistakenly thought it was an asynchronous request.
- For synchronous requests, this collection is not needed to store unresponsive requests.
- Or think about it from another angle, where can dead letter events be stored?
my thoughts:
1. If there are performance requirements, I recommend removing `retryer` and using `ScheduledExecutorService` to implement asynchronous requests.
2. Continue to use `retryer` and remove the `UN_ACK_REQ` collection
3. If there is no need to retransmit, you can directly remove this function.
Do you have any better suggestions in this regard?
---
如果是异步请求就可能会出现飞书未能及时响应,而connectorServer开始重传的情况。这可能会导致飞书重复消费。
但是目前我的实现逻辑存在问题,因为使用`com.github.rholder.retry.Retryer`的方式属于同步请求,而我误以为是异步请求。
- 对于同步请求来说,并不需要这种集合来存储未响应的请求。
- 或者可以换另一个角度思考,这里可以存储死信事件?
我的想法:
1. 有性能需求的话,我建议移除`retryer`,自行使用`ScheduledExecutorService`实现异步请求。
2. 继续使用`retryer`,移除`UN_ACK_REQ`集合
3. 没必要重传的话,可以直接移除该功能。
请问您在这方面有什么更好的建议吗?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413516292
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
Lark's access credentials have an expiration mechanism, but the expiration time can be refreshed by re-obtaining new credentials within half an hour before expiration. The documentation provided by Lark is as follows:
> The maximum validity period of tenant_access_token is 2 hours. If this interface is called when the validity period is less than 30 minutes, a new tenant_access_token will be returned, and there will be two valid tenant_access_tokens at the same time.
---
飞书的访问凭证存在过期机制,但是在过期前半个小时内重新获取新的凭证即可刷新过期时间。飞书提供的文档说明如下:
> tenant_access_token 的最大有效期是 2 小时。如果在有效期小于 30 分钟的情况下,调用本接口,会返回一个新的 tenant_access_token,这会同时存在两个有效的 tenant_access_token。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413610328
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
1. For API documentation, see [here](https://open.larksuite.com/document/server-docs/getting-started/api-access-token/auth-v3/tenant_access_token_internal)
2. The following code will use the sdk provided by lark to request access credentials.
```java
TenantAccessTokenResp resp = client.ext().getTenantAccessTokenBySelfBuiltApp(
SelfBuiltTenantAccessTokenReq.newBuilder()
.appSecret(appSecret)
.appId(appId)
.build());
3. I don’t understand what reliability means. I hope you can give me a more detailed answer.
---
1. API文档参见[这里](https://open.feishu.cn/document/server-docs/authentication-management/access-token/tenant_access_token_internal)
2. 下面这段代码将使用飞书提供的sdk请求获取访问凭证
```java
TenantAccessTokenResp resp = client.ext().getTenantAccessTokenBySelfBuiltApp(
SelfBuiltTenantAccessTokenReq.newBuilder()
.appSecret(appSecret)
.appId(appId)
.build());
3. 我不明白可靠性是指什么,希望您能给我更详细的解答。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
1. For API documentation, see [here](https://open.larksuite.com/document/server-docs/getting-started/api-access-token/auth-v3/tenant_access_token_internal)
2. The following code will use the sdk provided by lark to request access credentials.
```java
TenantAccessTokenResp resp = client.ext().getTenantAccessTokenBySelfBuiltApp(
SelfBuiltTenantAccessTokenReq.newBuilder()
.appSecret(appSecret)
.appId(appId)
.build());
3. I don’t understand what reliability means. I hope you can give me a more detailed answer.
---
1. API文档参见[这里](https://open.feishu.cn/document/server-docs/authentication-management/access-token/tenant_access_token_internal)
2. 下面这段代码将使用飞书提供的sdk请求获取访问凭证
```java
TenantAccessTokenResp resp = client.ext().getTenantAccessTokenBySelfBuiltApp(
SelfBuiltTenantAccessTokenReq.newBuilder()
.appSecret(appSecret)
.appId(appId)
.build());
3. 我不明白可靠性是指什么,希望您能给我更详细的解答。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1416459332
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
It's okay, I haven't started this work yet. Thank you for your attention to this PR, I have learned a lot.
没关系,我还没有开始这项工作。感谢您对这个pr的关注,让我学到了很多东西。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415740183
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
> Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
What?! Thank you for correcting me!
@yanrongzhen Previously, in one of your PR, I made a mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413589156
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> The maximum validity period of tenant_access_token is 2 hours. If this interface is called when the validity period is less than 30 minutes, a new tenant_access_token will be returned,
Which interface does this refer to? If there is a possibility of not calling this interface for a long time, I think it is possible to set a scheduled credential refresh task with an interval of less than 2 hours to improve reliability.
这个接口是指哪个?如果有可能出现长时间不调用该接口,我觉得可以设置一个间隔小于2小时的定时刷新凭证的机制提升可靠性。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> The maximum validity period of tenant_access_token is 2 hours. If this interface is called when the validity period is less than 30 minutes, a new tenant_access_token will be returned,
Which interface does this refer to? If there is a possibility of not calling this interface for a long time, I think it is possible to set a scheduled credential refresh task with an interval of less than 2 hours to improve reliability.
这个接口是指哪个?如果有可能出现长时间不调用该接口,我觉得可以设置一个间隔小于2小时的定时刷新凭证的机制提升可靠性。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415845536
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
Yes,i make a mistake.I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415694862
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
+ } else {
+ Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
Review Comment:
This is my question: Is this exception thrown internally by om.github.rholder.retry.Retryer?
这是我的疑问:该异常是com.github.rholder.retry.Retryer内部抛出来的吗?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413380269
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
For the management of access credentials, I actually plan to make it a global configuration. Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
(Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?)
---
对于访问凭证的管理,我其实打算做成一个全局配置,其他服务可能也会需要用到凭证管理。如果放进ImServiceHandler,感觉似乎只是为了ImServiceHandler而服务。不过我也有可能过度设计了。
(其实也有一个点令我疑惑,如果访问凭证只有一种,何必使用Cache集合呢?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413405564
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/ConnectRecordExtensionKeys.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark;
+
+/**
+ * Constants of record extension key.
+ */
+public interface ConnectRecordExtensionKeys {
+
+ /**
+ * {@code text} or {@code markdown}, otherwise use {@code text} to replace.
+ */
+ String TEMPLATE_TYPE_4_LARK = "templatetype4lark";
+
+ /**
+ * The value format is {@code id,name;id,name;}. <br/>
Review Comment:
https://github.com/apache/eventmesh/actions/runs/7080866475/job/19270211624
Java 11 does not support such doc code `<br/>`, and `<p/>` should also not.
Java 11 不支持这样的doc代码`<br/>` ,`<p/>`应该也不行。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Thank you. I am somewhat disappointed with my review work in PR #4522.
谢谢。我对自己在PR #4522的review工作感到有点失望。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/ConnectRecordExtensionKeys.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark;
+
+/**
+ * Constants of record extension key.
+ */
+public interface ConnectRecordExtensionKeys {
+
+ /**
+ * {@code text} or {@code markdown}, otherwise use {@code text} to replace.
+ */
+ String TEMPLATE_TYPE_4_LARK = "templatetype4lark";
+
+ /**
+ * The value format is {@code id,name;id,name;}. <br/>
Review Comment:
https://github.com/apache/eventmesh/actions/runs/7080866475/job/19270211624
Java 11 does not support such doc code `<br/>`, and `<p/>` should also not.
Java 11 不支持这样的doc代码`<br/>` ,`<p/>`应该也不行。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Thank you. I am somewhat disappointed with my review work in PR #4522.
谢谢。我对自己在PR #4522的review工作感到有点失望。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
If it is possible to develop other services that require this credential to further enrich the connector, you can leave a note in this method for other pending services that may require this credential.
如果还有可能开发其他需要该凭证的服务来进一步丰富该Connector,您可以在该方法备注一下可能需要该凭证的其他等待实现的服务。
> Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?
I encountered similar doubts in an other connector before https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091, and the developer explained that because credentials will expire and new credential will be constantly obtained in the code https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666. I'm not sure if the credential here will expire. If it do, not only does it require Cache, but also a mechanism for regular updates.
我之前在其他Connector遇到类似的疑惑 https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091 ,开发者解释到因为凭证会过期 https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666 ,并且在代码中会不断获取新的凭证。我不清楚这里的凭证会不会过期,如果会过期,不仅需要Cache,还需要定时更新的机制。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
> The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
Could you explain what incorrect phenomenon or logic may occur when retry, which may lead to "repeated consumption by Feishu"? Because I don't understand the difference between retrying and "repeated consumption by Feishu".
能否说明一下在重试的时候“会导致飞书重复消费”会出现什么不正确的现象或者逻辑?因为我没明白重试和“飞书重复消费”有什么区别。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
If it is possible to develop other services that require this credential to further enrich the connector, you can leave a note in this method for other pending services that may require this credential.
如果还有可能开发其他需要该凭证的服务来进一步丰富该Connector,您可以在该方法备注一下可能需要该凭证的其他等待实现的服务。
> Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?
I encountered similar doubts in an other connector before https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091, and the developer explained that because credentials will expire and new credential will be constantly obtained in the code https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666. I'm not sure if the credential here will expire. If it do, not only does it require Cache, but also a mechanism for regular updates.
我之前在其他Connector遇到类似的疑惑 https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091 ,开发者解释到因为凭证会过期 https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666 ,并且在代码中会不断获取新的凭证。我不清楚这里的凭证会不会过期,如果会过期,不仅需要Cache,还需要定时更新的机制。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
> The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
Could you explain what incorrect phenomenon or logic may occur when retry, which may lead to "repeated consumption by Feishu"? Because I don't understand the difference between retrying and "repeated consumption by Feishu".
能否说明一下在重试的时候“会导致飞书重复消费”会出现什么不正确的现象或者逻辑?因为我没明白重试和“飞书重复消费”有什么区别。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413409355
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
If it is possible to develop other services that require this credential to further enrich the connector, you can leave a note in this method for other pending services that may require this credential.
如果还有可能开发其他需要该凭证的服务来进一步丰富该Connector,您可以在该方法备注一下可能需要该凭证的其他等待实现的服务。
> Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?
I encountered similar doubts in an other connector before https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091, and the developer explained that because credentials will expire and new credential will be constantly obtained in the code https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666. I'm not sure if the credential here will expire. If it do, not only does it require Cache, but also a mechanism for regular updates.
我之前在其他Connector遇到类似的疑惑 https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091 ,开发者解释到因为凭证会过期 https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666 ,并且在代码中会不断获取新的凭证。我不清楚这里的凭证会不会过期,如果会过期,不仅需要Cache,还需要定时更新的机制。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
> Other services may also need to use the credentials management . If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
If it is possible to develop other services that require this credential to further enrich the connector, you can leave a note in this method for other pending services that may require this credential.
如果还有可能开发其他需要该凭证的服务来进一步丰富该Connector,您可以在该方法备注一下可能需要该凭证的其他等待实现的服务。
> Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?
I encountered similar doubts in an other connector before https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091, and the developer explained that because credentials will expire and new credential will be constantly obtained in the code https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666. I'm not sure if the credential here will expire. If it do, not only does it require Cache, but also a mechanism for regular updates.
我之前在其他Connector遇到类似的疑惑 https://github.com/apache/eventmesh/pull/4594#discussion_r1412702091 ,开发者解释到因为凭证会过期 https://github.com/apache/eventmesh/pull/4594#discussion_r1412803666 ,并且在代码中会不断获取新的凭证。我不清楚这里的凭证会不会过期,如果会过期,不仅需要Cache,还需要定时更新的机制。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838020382
I have question about why you change the module name from Feishu to lark ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1416382025
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I‘m sorry for my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。我对自己的反反复复感到抱歉。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I‘m sorry for my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。我对自己的反反复复感到抱歉。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413598725
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
I think there is still a need for retransmission when the response fails.
Based on your idea, I think it's possible to add a configuration for asynchronous retry. When synchronizing, continue to use retrier and remove UN_ ACK_ REQ set. When asynchronous, use Scheduling ExecutorService to implement asynchronous requests. If you agree, it is also possible to simplify the process first and further improve it in the new PR.
我觉得响应失败时的重传还是有需要的。
根据您的想法,我觉得可以增加是否异步重试的配置。同步时,继续使用retryer,移除UN_ACK_REQ集合。异步时,使用ScheduledExecutorService实现异步请求。如果您赞同的话,先简化处理,以后在新的PR进一步完善,也是可以的。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
I think there is still a need for retransmission when the response fails.
Based on your idea, I think it's possible to add a configuration for asynchronous retry. When synchronizing, continue to use retrier and remove UN_ ACK_ REQ set. When asynchronous, use Scheduling ExecutorService to implement asynchronous requests. If you agree, it is also possible to simplify the process first and further improve it in the new PR.
我觉得响应失败时的重传还是有需要的。
根据您的想法,我觉得可以增加是否异步重试的配置。同步时,继续使用retryer,移除UN_ACK_REQ集合。异步时,使用ScheduledExecutorService实现异步请求。如果您赞同的话,先简化处理,以后在新的PR进一步完善,也是可以的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413401314
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Thank you. I am somewhat disappointed with my review work in PR #4522.
谢谢。我对自己在PR #4522的review工作感到有点失望。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415767215
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
At present, I have only simulated the scenario where the Lark server responds incorrectly, and used `verify` to verify that after passing the retry mechanism, the `create` method was indeed called the corresponding number of times (here it is 3*(3+1) times) . If there is a better way for testing, please give me some hints.
目前我只模拟了lark服务端响应错误的场景,并通过`verify`来验证通过重试机制后,确实调用了`create`方发相应的次数(在此是3*(3+1)次)。如果对于测试有更好办法的话,请给我一些提示。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
At present, I have only simulated the scenario where the Lark server responds incorrectly, and used `verify` to verify that after passing the retry mechanism, the `create` method was indeed called the corresponding number of times (here it is 3*(3+1) times) . If there is a better way for testing, please give me some hints.
目前我只模拟了lark服务端响应错误的场景,并通过`verify`来验证通过重试机制后,确实调用了`create`方发相应的次数(在此是3*(3+1)次)。如果对于测试有更好办法的话,请给我一些提示。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
At present, I have only simulated the scenario where the Lark server responds incorrectly, and used `verify` to verify that after passing the retry mechanism, the `create` method was indeed called the corresponding number of times (here it is 3*(3+1) times) . If there is a better way for testing, please give me some hints.
目前我只模拟了飞书服务端响应错误的场景,并通过`verify`来验证通过重试机制后,确实调用了`create`方发相应的次数(在此是3*(3+1)次)。如果对于测试有更好办法的话,请给我一些提示。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415832718
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
Oh! I missed the last line of code. retrySink() is already sufficient for me. This review should be written under regularSink().
Oh! 我错过了最后一行代码,retrySink() 已经可以了。这段review应该写在regularSink()下。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
There is no hidden dangers, just more efficiency. Now, for each data send, create a Runnable task and submit it to the thread pool. If you directly put the data or some failed data into the queue, you only need to create a Runnable task to submit to the thread pool, and then loop through the task to retrieve the data. If there are many tasks in the thread pool and processing time is insufficient, there may be frequent switching of thread processing tasks after a certain accumulation. This is my opinion, what do you think?
没有隐藏的不安全,只是更高效。现在每发一次数据,创建一个Runnable任务,然后提交到线程池。如果直接把数据或部分失败数据放到队列里,只需要创建一个Runnable任务提交给线程池,然后任务中循环取数据。如果线程池中任务较多且处理不及时时,有一定累积后,可能会频繁切换线程处理任务。这是我的观点,您怎么看?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
There is no hidden dangers, just more efficiency. Now, for each data send, create a Runnable task and submit it to the thread pool. If you directly put the data or some failed data into the queue, you only need to create a Runnable task to submit to the thread pool, and then loop through the task to retrieve the data. If there are many tasks in the thread pool and processing time is insufficient, there may be frequent switching of thread processing tasks after a certain accumulation. This is my opinion, what do you think?
没有隐藏的不安全,只是更高效。现在每发一次数据,创建一个Runnable任务,然后提交到线程池。如果直接把数据或部分失败数据放到队列里,只需要创建一个Runnable任务提交给线程池,然后任务中循环取数据。如果线程池中任务较多且处理不及时时,有一定累积后,可能会频繁切换线程处理任务。这是我的观点,您怎么看?
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
Oh! I missed the last line of code. retrySink() is already sufficient for me. This review should be written under regularSink().
Oh! 我错过了最后一行代码,retrySink() 已经可以了。这段review应该写在regularSink()下。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415996443
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。希望您不要介意我的反反复复。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415674144
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
It seems unnecessary to add this annotation because Strictness.LENIENT is the default execution mode of Mockito.
似乎没有必要增加该注解,因为Strictness.LENIENT是Mockito默认的执行模式。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
In this unit test class, the effectiveness (I don't know if the term is appropriate, meaning there is no checksum assertion) is relatively weak for both situations where there is no retry and asynchronous retry. It would be best if optimization could continue in the future.
该单元测试类中,对于不会重试的情况和异步重试的情况的测试,有效性(我不知道这个词是否恰当,意思是没有校验和断言)都比较弱。如果后续能继续优化那最好不过。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
Can the asynchronous mechanism in this method be taken outside of the sinkAsync() method? For example, storing failed messages or each message in a queue, and then asynchronously retry or process the messages in the queue. This way, there is no need to submit a task to the thread pool every time a message is sent. If you agree but want to continue optimizing in the new PR, you can also do so.
能否将该方法中的异步机制拿到sinkAsync()方法外面?比如将发送失败的消息或者每个消息先存到一个队列中,然后对队列中的消息进行异步重试或者异步处理。这样就不用每发一个消息都向线程池提交一次任务。如果您赞同但想在新的PR中继续优化,也可以。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
+ } else {
+ Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
Review Comment:
This is my question: Is this exception thrown internally by om.github.rholder.retry.Retryer?
这是我的疑问:该异常是com.github.rholder.retry.Retryer内部抛出来的?
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
It seems unnecessary to add this annotation because Strictness.LENIENT is the default execution mode of Mockito.
似乎没有必要增加该注解,因为Strictness.LENIENT是Mockito默认的执行模式。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
In this unit test class, the effectiveness (I don't know if the term is appropriate, meaning there is no checksum assertion) is relatively weak for both situations where there is no retry and asynchronous retry. It would be best if optimization could continue in the future.
该单元测试类中,对于不会重试的情况和异步重试的情况的测试,有效性(我不知道这个词是否恰当,意思是没有校验和断言)都比较弱。如果后续能继续优化那最好不过。
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
+ } else {
+ Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
Review Comment:
This is my question: Is this exception thrown internally by om.github.rholder.retry.Retryer?
这是我的疑问:该异常是com.github.rholder.retry.Retryer内部抛出来的?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
Can the asynchronous mechanism in this method be taken outside of the sinkAsync() method? For example, storing failed messages or each message in a queue, and then asynchronously retry or process the messages in the queue. This way, there is no need to submit a task to the thread pool every time a message is sent. If you agree but want to continue optimizing in the new PR, you can also do so.
能否将该方法中的异步机制拿到sinkAsync()方法外面?比如将发送失败的消息或者每个消息先存到一个队列中,然后对队列中的消息进行异步重试或者异步处理。这样就不用每发一个消息都向线程池提交一次任务。如果您赞同但想在新的PR中继续优化,也可以。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415975391
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。希望您不要介意我的反反复复。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。希望您不要介意我的反反复复。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。希望您不要介意我的反反复复。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1416459332
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
It's okay, I haven't started this work yet. Thank you for your review of this PR, I learned a lot.
没关系,我还没有开始这项工作。感谢您对这个pr的review,让我学到了很多东西。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1837761955
## [Codecov](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `89 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`5998926`)](https://app.codecov.io/gh/apache/eventmesh/commit/5998926750cc2f77625a0bdfe82be5838dab28db?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 16.79% compared to head [(`5956b4e`)](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.10%.
> Report is 2 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ventmesh/connector/lark/sink/ImServiceHandler.java](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1sYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2xhcmsvc2luay9JbVNlcnZpY2VIYW5kbGVyLmphdmE=) | 51.30% | [49 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...nnector/lark/sink/connector/LarkSinkConnector.java](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1sYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2xhcmsvc2luay9jb25uZWN0b3IvTGFya1NpbmtDb25uZWN0b3IuamF2YQ==) | 54.54% | [18 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...onnector/lark/sink/config/SinkConnectorConfig.java](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1sYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2xhcmsvc2luay9jb25maWcvU2lua0Nvbm5lY3RvckNvbmZpZy5qYXZh) | 66.66% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...tmesh/connector/lark/server/LarkConnectServer.java](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1sYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2xhcmsvc2VydmVyL0xhcmtDb25uZWN0U2VydmVyLmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...connector/lark/config/LarkConnectServerConfig.java](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1sYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2xhcmsvY29uZmlnL0xhcmtDb25uZWN0U2VydmVyQ29uZmlnLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4599 +/- ##
============================================
+ Coverage 16.79% 17.10% +0.30%
- Complexity 1677 1711 +34
============================================
Files 782 791 +9
Lines 29317 29551 +234
Branches 2532 2537 +5
============================================
+ Hits 4924 5054 +130
- Misses 23933 24023 +90
- Partials 460 474 +14
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/eventmesh/pull/4599?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413374636
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
- The main function of this collection is to track requests that do not return a response.
- The interface exposed by the Lark platform can guarantee us that if the UUID is the same, the message will not be sent repeatedly.
- A UUID will be injected inside `CreateMessageReq`. The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
---
- 这个集合的主要作用是追踪那些未返回响应的请求。
- 飞书平台暴露的接口可以向我们保证——如果UUID一样,那么消息就不会重复发送。
- 在`CreateMessageReq`内部将会注入UUID,原先的代码会在每次重传时,都重新注入一个新的UUID,这将会导致飞书重复消费。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413380269
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
For the management of access credentials, I actually plan to make it a global configuration. Other services may also need to use the credentials. If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
(Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?)
---
对于访问凭证的管理,我其实打算做成一个全局配置,其他服务可能也会需要用到凭证。如果放进ImServiceHandler,感觉似乎只是为了ImServiceHandler而服务。不过我也有可能过度设计了。
(其实也有一个点令我疑惑,如果访问凭证只有一种,何必使用Cache集合呢?)
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
For the management of access credentials, I actually plan to make it a global configuration. Other services may also need to use the credentials. If you put ImServiceHandler in, it feels like it is only serving ImServiceHandler. But it's also possible that I'm over-designing it.
(Actually, there is one point that puzzles me. If there is only one access credential, why use the Cache collection?)
---
对于访问凭证的管理,我其实打算做成一个全局配置,其他服务可能也会需要用到凭证。如果放进ImServiceHandler,感觉似乎只是为了ImServiceHandler而服务。不过我也有可能过度设计了。
(其实也有一个点令我疑惑,如果访问凭证只有一种,何必使用Cache集合呢?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413376877
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413467635
##########
eventmesh-connectors/eventmesh-connector-lark/README.md:
##########
@@ -0,0 +1,69 @@
+# eventmesh-connector-lark
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838245471
@hhuang1231
The Feishu document said that "Feishu and Lark are independent products operated by different entities under ByteDance company. Feishu accounts can only be used in the Feishu client, and the Feishu client only supports the use of Feishu accounts."
If you want to be compatible with both products, you may need to pay attention to the interoperability of the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413632805
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
What I mean by reliability is that there will be no situation where credentials expire and message sending fails when using this Sink. From the code perspective, the credential is only obtained once when Sink is started. Therefore, I think it is possible to set a mechanism for timed credential refresh with an interval of less than 2 hours.
我说的可靠性是指不会在使用该Sink的时候出现凭证过期导致消息发送失败的情况。从代码来看,该凭证只在启动Sink的时候获取一次,那我觉得可以设置一个间隔小于2小时的定时刷新凭证的机制。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
What I mean by reliability is that there will be no situation where credentials expire and message sending fails when using this Sink. From the code perspective, the credential is only obtained once when Sink is started. Therefore, I think it is possible to set a mechanism for timed credential refresh with an interval of less than 2 hours.
我说的可靠性是指不会在使用该Sink的时候出现凭证过期导致消息发送失败的情况。从代码来看,该凭证只在启动Sink的时候获取一次,那我觉得可以设置一个间隔小于2小时的定时刷新凭证的机制。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413374636
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
- The main function of this collection is to track requests that do not return a response.
- The interface exposed by the Lark platform can guarantee us that if the UUID is the same, the message will not be sent repeatedly. A UUID will be injected inside `CreateMessageReq`.
- The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
- 这个集合的主要作用是追踪那些未返回响应的请求。
- 飞书平台暴露的接口可以向我们保证——如果UUID一样,那么消息就不会重复发送。
- 在`CreateMessageReq`内部将会注入UUID,原先的代码会在每次重传时,都重新注入一个新的UUID,这将会导致飞书重复消费。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
- The main function of this collection is to track requests that do not return a response.
- The interface exposed by the Lark platform can guarantee us that if the UUID is the same, the message will not be sent repeatedly. A UUID will be injected inside `CreateMessageReq`.
- The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
- 这个集合的主要作用是追踪那些未返回响应的请求。
- 飞书平台暴露的接口可以向我们保证——如果UUID一样,那么消息就不会重复发送。
- 在`CreateMessageReq`内部将会注入UUID,原先的代码会在每次重传时,都重新注入一个新的UUID,这将会导致飞书重复消费。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1840612423
@pandaapo PLAT, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415740183
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1416382025
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). If you haven't made any changes here yet, I think we should adopt your current way for now. I‘m sorry for my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于您现在的方案(每个消息给线程池提交一个任务)。如果您还没有修改,我觉得就先采用您目前的方案。我对自己的反反复复感到抱歉。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413421859
##########
eventmesh-connectors/eventmesh-connector-lark/README.md:
##########
@@ -0,0 +1,69 @@
+# eventmesh-connector-lark
Review Comment:
You can split the document into README.md and README_ CN.md
您可以将该文档拆成README.md和README_CN.md。
##########
eventmesh-connectors/eventmesh-connector-lark/README.md:
##########
@@ -0,0 +1,69 @@
+# eventmesh-connector-lark
Review Comment:
You can split the document into README.md and README_ CN.md
您可以将该文档拆成README.md和README_CN.md。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838068433
> > I have question about why you change the module name from Feishu to lark ?
>
> @xwm1992 I found that the dingding module was changed to dingtalk.So I think it is necessary to change Feishu to lark.
DingDing was changed to DingTalk because the English name of the product itself is called DingTalk, and Feishu was changed to lark. I don’t know if it’s still a product or an overseas version of Feishu. @hhuang1231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415767215
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandlerTest.java:
##########
@@ -85,31 +88,75 @@ public void setup() throws Exception {
@Test
public void testRegularSink() throws Exception {
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ regularSink();
+ }
+
+ @Test
+ public void testRegularSinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ regularSink();
+ }
+
+ private void regularSink() throws Exception {
final int times = 3;
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- imServiceHandler.sink(connectRecord);
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ long duration = retryDelayInMills * times;
+ Thread.sleep(duration);
+ } else {
+ imServiceHandler.sink(connectRecord);
+ }
}
-
- verify(message, times(times)).create(any(), any());
}
@Test
public void testRetrySink() throws Exception {
- doThrow(new Exception()).when(message).create(any(), any());
+ sinkConnectorConfig.setSinkAsync("false");
+ init();
+ retrySink();
+ }
+
+ @Test
+ public void testRetrySinkAsync() throws Exception {
+ sinkConnectorConfig.setSinkAsync("true");
+ init();
+ retrySink();
+ }
+
+ private void retrySink() throws Exception {
+ CreateMessageResp resp = new CreateMessageResp();
+ resp.setCode(1);
+ doReturn(resp).when(message).create(any(), any());
final int times = 3;
+ long retryDelayInMills = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes());
+ // (maxRetryTimes + 1) event are actually sent
+ int sinkTimes = times * (maxRetryTimes + 1);
+ long duration = retryDelayInMills * sinkTimes;
+
for (int i = 0; i < times; i++) {
RecordPartition partition = new RecordPartition();
RecordOffset offset = new RecordOffset();
ConnectRecord connectRecord = new ConnectRecord(partition, offset,
System.currentTimeMillis(), "test-lark".getBytes(StandardCharsets.UTF_8));
- Assertions.assertThrows(Exception.class, () -> imServiceHandler.sink(connectRecord));
- }
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ imServiceHandler.sinkAsync(connectRecord);
- // (maxRetryTimes + 1) event are actually sent
- verify(message, times(times * (Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1))).create(any(), any());
+ Thread.sleep(duration);
Review Comment:
At present, I have only simulated the scenario where the Lark server responds incorrectly, and used `verify` to verify that after passing the retry mechanism, the `create` method was indeed called the corresponding number of times (here it is 3*(3+1) times) . If there is a better way for testing, please give me some hints.
目前我只模拟了飞书服务端响应错误的场景,并通过`verify`来验证通过重试机制后,确实调用了`create`方法相应的次数(在此是3*(3+1)次)。如果对于测试有更好办法的话,请给我一些提示。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415755336
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
> Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
What?! Thank you for correcting me!
@yanrongzhen Previously, in your PR #4599, I made a mistake. Sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415975391
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。希望您不要介意我的反反复复。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
When I realized that this was sending messages to Lark, I thought that in practical applications, most scenarios may be used as event notifications, so there may not be the transmission of a large number of messages between message middleware. When considering this, my suggestion (to submit a task to the thread pool and process messages in a loop within the task) may not be more efficient than your current solution (to submit a task for each message to the thread pool). I hope you don't mind my repeated actions.
当我意识到这是给飞书发消息时,我想实际应用中,可能大多数场景是用来作为一种事件通知,所以也许并没有比如消息中间件之间那种大量消息的传输。这样考虑时,我的建议中(给线程池提交一个任务,在任务中循环处理消息)的效率也许并不优于没有您现在的方案(每个消息给线程池提交一个任务)。希望您不要介意我的反反复复。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415740183
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413329673
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Have you noticed that Feishu Sink was able to function normally when there were no issues here before?
您是否注意到之前Feishu Sink当这里没有问题的时候能否正常工作?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Have you noticed that Feishu Sink was able to function normally when there were no issues here before?
您是否注意到之前Feishu Sink当这里没有问题的时候能否正常工作?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
Review Comment:
You can directly annotate the `@Deprecated` annotation onto Connector#init() method.
可以直接将`@Deprecated`注解标注到Connector#init()方法上。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
Can this method be directly defined in ImServiceHandler?
能否直接将该方法定义在ImServiceHandler中?
##########
eventmesh-connectors/eventmesh-connector-lark/build.gradle:
##########
@@ -15,19 +15,20 @@
* limitations under the License.
*/
-List feishu = [
- "com.larksuite.oapi:oapi-sdk:$feishu_version",
+List lark = [
+ "com.larksuite.oapi:oapi-sdk:$lark_version",
"com.github.rholder:guava-retrying:$guava_retrying_version",
"org.apache.httpcomponents:httpclient",
project(":eventmesh-common")
]
dependencies {
api project(":eventmesh-openconnect:eventmesh-openconnect-java")
- implementation feishu
+ implementation lark
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
- testImplementation "org.mockito:mockito-core"
+ //testImplementation "org.mockito:mockito-core"
Review Comment:
Remove unnecessary dependencies directly
不需要的依赖就直接移除吧。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
I didn't understand the purpose of this Map, could you explain it?
没看明白这个集合的作用,能否解释一下?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
+
+ private SinkConnectorConfig sinkConnectorConfig;
+
+ private ImService imService;
+
+ private RequestOptions requestOptions;
+
+ private Retryer<ConnectRecord> retryer;
+
+ public ImServiceHandler() {}
+
+ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
+ ImServiceHandler imServiceHandler = new ImServiceHandler();
+ imServiceHandler.sinkConnectorConfig = sinkConnectorConfig;
+ imServiceHandler.imService = Client.newBuilder(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret())
+ .httpTransport(new OkHttpTransport(new OkHttpClient().newBuilder()
+ .callTimeout(3L, TimeUnit.SECONDS)
+ .build())
+ )
+ .disableTokenCache()
+ .requestTimeout(3, TimeUnit.SECONDS)
+ .build()
+ .im();
+
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
+
+ imServiceHandler.requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
+ if (times == maxRetryTimes) {
+ UN_ACK_REQ.remove(attempt.get());
+ }
+ }
+ })
+ .build();
+ return imServiceHandler;
+ }
+
+ public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
+ CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
+ .receiveId(sinkConnectorConfig.getReceiveId())
+ .uuid(UUID.randomUUID().toString());
+
+ String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
+ if (null == templateTypeKey || "null".equals(templateTypeKey)) {
+ templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
+ }
+ LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
+ if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
+ bodyBuilder.content(createTextContent(connectRecord))
+ .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
+ } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
+ String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
+ .orElse("EventMesh-Message");
+ bodyBuilder.content(createInteractiveContent(connectRecord, title))
+ .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
+ }
+
+ return CreateMessageReq.newBuilder()
+ .receiveIdType(sinkConnectorConfig.getReceiveIdType())
+ .createMessageReqBody(bodyBuilder.build())
+ .build();
+ });
+ CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
+ if (resp.getCode() != 0) {
+ log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
+ return connectRecord;
+ }
+
+ UN_ACK_REQ.remove(connectRecord);
+ return null;
+ });
+ }
+
+ private String createTextContent(ConnectRecord connectRecord) {
+ MessageText.Builder msgBuilder = MessageText.newBuilder();
+
+ if (needAtAll(connectRecord)) {
+ msgBuilder.atAll();
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ msgBuilder.atUser(kv[0], kv[1]);
+ }
+ }
+
+ String escapedString = StringEscapeUtils.escapeJava(new String((byte[]) connectRecord.getData()));
+ return msgBuilder.text(escapedString).build();
+ }
+
+ private String createInteractiveContent(ConnectRecord connectRecord, String title) {
+ StringBuilder sb = new StringBuilder();
+ if (needAtAll(connectRecord)) {
+ atAll(sb);
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ atUser(sb, kv[0]);
+ }
+ }
+ sb.append(new String((byte[]) connectRecord.getData()));
+
+ MessageCardConfig config = MessageCardConfig.newBuilder()
+ .enableForward(true)
+ .wideScreenMode(true)
+ .updateMulti(true)
+ .build();
+
+ // cardUrl, Only in need of PC, mobile side jump different links to use
+ /*MessageCardURL cardURL = MessageCardURL.newBuilder()
+ .pcUrl("http://www.baidu.com")
+ .iosUrl("http://www.google.com")
+ .url("http://open.feishu.com")
+ .androidUrl("http://www.jianshu.com")
+ .build();*/
Review Comment:
Can this comment and the commented out code be removed?
这段注释和注释掉的代码能否移除掉?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
Review Comment:
You can directly annotate the `@Deprecated` annotation onto Connector#init() method.
可以直接将`@Deprecated`注解标注到Connector#init()方法上。
##########
eventmesh-connectors/eventmesh-connector-lark/build.gradle:
##########
@@ -15,19 +15,20 @@
* limitations under the License.
*/
-List feishu = [
- "com.larksuite.oapi:oapi-sdk:$feishu_version",
+List lark = [
+ "com.larksuite.oapi:oapi-sdk:$lark_version",
"com.github.rholder:guava-retrying:$guava_retrying_version",
"org.apache.httpcomponents:httpclient",
project(":eventmesh-common")
]
dependencies {
api project(":eventmesh-openconnect:eventmesh-openconnect-java")
- implementation feishu
+ implementation lark
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
- testImplementation "org.mockito:mockito-core"
+ //testImplementation "org.mockito:mockito-core"
Review Comment:
Remove unnecessary dependencies directly
不需要的依赖就直接移除吧。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
+
+ private SinkConnectorConfig sinkConnectorConfig;
+
+ private ImService imService;
+
+ private RequestOptions requestOptions;
+
+ private Retryer<ConnectRecord> retryer;
+
+ public ImServiceHandler() {}
+
+ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
+ ImServiceHandler imServiceHandler = new ImServiceHandler();
+ imServiceHandler.sinkConnectorConfig = sinkConnectorConfig;
+ imServiceHandler.imService = Client.newBuilder(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret())
+ .httpTransport(new OkHttpTransport(new OkHttpClient().newBuilder()
+ .callTimeout(3L, TimeUnit.SECONDS)
+ .build())
+ )
+ .disableTokenCache()
+ .requestTimeout(3, TimeUnit.SECONDS)
+ .build()
+ .im();
+
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
+
+ imServiceHandler.requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
+ int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
+ if (times == maxRetryTimes) {
+ UN_ACK_REQ.remove(attempt.get());
+ }
+ }
+ })
+ .build();
+ return imServiceHandler;
+ }
+
+ public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
+ CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
+ .receiveId(sinkConnectorConfig.getReceiveId())
+ .uuid(UUID.randomUUID().toString());
+
+ String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
+ if (null == templateTypeKey || "null".equals(templateTypeKey)) {
+ templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
+ }
+ LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
+ if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
+ bodyBuilder.content(createTextContent(connectRecord))
+ .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
+ } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
+ String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
+ .orElse("EventMesh-Message");
+ bodyBuilder.content(createInteractiveContent(connectRecord, title))
+ .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
+ }
+
+ return CreateMessageReq.newBuilder()
+ .receiveIdType(sinkConnectorConfig.getReceiveIdType())
+ .createMessageReqBody(bodyBuilder.build())
+ .build();
+ });
+ CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
+ if (resp.getCode() != 0) {
+ log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
+ return connectRecord;
+ }
+
+ UN_ACK_REQ.remove(connectRecord);
+ return null;
+ });
+ }
+
+ private String createTextContent(ConnectRecord connectRecord) {
+ MessageText.Builder msgBuilder = MessageText.newBuilder();
+
+ if (needAtAll(connectRecord)) {
+ msgBuilder.atAll();
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ msgBuilder.atUser(kv[0], kv[1]);
+ }
+ }
+
+ String escapedString = StringEscapeUtils.escapeJava(new String((byte[]) connectRecord.getData()));
+ return msgBuilder.text(escapedString).build();
+ }
+
+ private String createInteractiveContent(ConnectRecord connectRecord, String title) {
+ StringBuilder sb = new StringBuilder();
+ if (needAtAll(connectRecord)) {
+ atAll(sb);
+ }
+ String atUsers = needAtUser(connectRecord);
+ if (!atUsers.isEmpty()) {
+ String[] users = atUsers.split(";");
+
+ for (String user : users) {
+ String[] kv = user.split(",");
+ atUser(sb, kv[0]);
+ }
+ }
+ sb.append(new String((byte[]) connectRecord.getData()));
+
+ MessageCardConfig config = MessageCardConfig.newBuilder()
+ .enableForward(true)
+ .wideScreenMode(true)
+ .updateMulti(true)
+ .build();
+
+ // cardUrl, Only in need of PC, mobile side jump different links to use
+ /*MessageCardURL cardURL = MessageCardURL.newBuilder()
+ .pcUrl("http://www.baidu.com")
+ .iosUrl("http://www.google.com")
+ .url("http://open.feishu.com")
+ .androidUrl("http://www.jianshu.com")
+ .build();*/
Review Comment:
Can this comment and the commented out code be removed?
这段注释和注释掉的代码能否移除掉?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
I didn't understand the purpose of this Map, could you explain it?
没看明白这个集合的作用,能否解释一下?
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/connector/LarkSinkConnector.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink.connector;
+
+import static org.apache.eventmesh.connector.lark.sink.ImServiceHandler.create;
+
+import org.apache.eventmesh.connector.lark.sink.ImServiceHandler;
+import org.apache.eventmesh.connector.lark.sink.config.LarkSinkConfig;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SinkConnectorContext;
+import org.apache.eventmesh.openconnect.api.sink.Sink;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.github.rholder.retry.RetryException;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.lark.oapi.Client;
+import com.lark.oapi.core.enums.AppType;
+import com.lark.oapi.core.request.SelfBuiltTenantAccessTokenReq;
+import com.lark.oapi.core.response.TenantAccessTokenResp;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class LarkSinkConnector implements Sink {
+
+ public static final String TENANT_ACCESS_TOKEN = "tenant_access_token";
+
+ /**
+ * replace lark build-in tokenCache
+ */
+ public static final Cache<String, String> AUTH_CACHE = CacheBuilder.newBuilder()
+ .initialCapacity(12)
+ .maximumSize(10)
+ .concurrencyLevel(5)
+ .expireAfterWrite(30, TimeUnit.MINUTES)
+ .build();
+
+ private LarkSinkConfig sinkConfig;
+
+ private ImServiceHandler imServiceHandler;
+
+ private final AtomicBoolean started = new AtomicBoolean(false);
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return LarkSinkConfig.class;
+ }
+
+ @Override
+ public void init(Config config) {
+ // Deprecated
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) {
+ // init config for lark sink connector
+ SinkConnectorContext sinkConnectorContext = (SinkConnectorContext) connectorContext;
+ this.sinkConfig = (LarkSinkConfig) sinkConnectorContext.getSinkConfig();
+
+ SinkConnectorConfig sinkConnectorConfig = sinkConfig.getSinkConnectorConfig();
+ sinkConnectorConfig.validateSinkConfiguration();
+
+ imServiceHandler = create(sinkConnectorConfig);
+ }
+
+ @Override
+ public void start() {
+ if (!started.compareAndSet(false, true)) {
+ log.info("LarkSinkConnector has been started.");
+ }
+ }
+
+ @Override
+ public void commit(ConnectRecord record) {
+ // Sink does not need to implement
+ }
+
+ @Override
+ public String name() {
+ return this.sinkConfig.getSinkConnectorConfig().getConnectorName();
+ }
+
+ @Override
+ public void stop() {
+ if (!started.compareAndSet(true, false)) {
+ log.info("LarkSinkConnector has not started yet.");
+ }
+ }
+
+ @Override
+ public void put(List<ConnectRecord> sinkRecords) {
+ for (ConnectRecord connectRecord : sinkRecords) {
+ try {
+ imServiceHandler.sink(connectRecord);
+ } catch (ExecutionException | RetryException e) {
+ log.error("Failed to sink event to lark", e);
+ }
+ }
+ }
+
+ @SneakyThrows
+ public static String getTenantAccessToken(String appId, String appSecret) {
Review Comment:
Can this method be directly defined in ImServiceHandler?
能否直接将该方法定义在ImServiceHandler中?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413391854
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
> The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
Could you explain what incorrect phenomenon or logic may occur when retry, which may lead to "repeated consumption by Feishu"? Because I don't understand the difference between retrying and "repeated consumption by Feishu".
能否说明一下在重试的时候“会导致飞书重复消费”会出现什么不正确的现象或者逻辑?因为我没明白重试和“飞书重复消费”有什么区别。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413410369
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Thank you. I am somewhat disappointed with my review work in PR #4522.
谢谢。我对自己在PR #4522的review工作感到有点失望。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/resources/sink-config.yml:
##########
@@ -17,14 +17,18 @@
pubSubConfig:
meshAddress: 127.0.0.1:10000
- subject: TopicTest
+ subject: TEST-TOPIC-LARK
idc: FT
env: PRD
- group: feishuSink
+ group: larkSink
appId: 5031
- userName: feishuSinkUser
- passWord: feishuPassWord
-connectorConfig:
- connectorName: feishuSink
- reciveId: reciveIdValue
- reciveType: open_id
+ userName: larkSinkUser
+ passWord: larkPassWord
+sinkConnectorConfig:
+ connectorName: larkSink
+ appId: appId
+ appSecret: appSecret
+ receiveIdType: open_id
+ receiveId: receiveId
Review Comment:
Thank you. I am somewhat disappointed with my review work in PR #4522.
谢谢。我对自己在PR #4522的review工作感到有点失望。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 merged PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413374636
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
- The main function of this collection is to track requests that do not return a response.
- The interface exposed by the Lark platform can guarantee us that if the UUID is the same, the message will not be sent repeatedly. A UUID will be injected inside `CreateMessageReq`.
- The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
---
- 这个集合的主要作用是追踪那些未返回响应的请求。
- 飞书平台暴露的接口可以向我们保证——如果UUID一样,那么消息就不会重复发送。
- 在`CreateMessageReq`内部将会注入UUID,原先的代码会在每次重传时,都重新注入一个新的UUID,这将会导致飞书重复消费。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#issuecomment-1838157814
> DingDing was changed to DingTalk because the English name of the product itself is called DingTalk, and Feishu was changed to lark. I don’t know if it’s still a same product or an overseas version of Feishu
@xwm1992 If the overseas Lark account I registered is correct, then the incident can be successfully sinked after testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413637683
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
ok, I will continue to implement asynchronous requests in this PR according to your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1413408941
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
> The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
Could you explain what incorrect phenomenon or logic may occur when retry, which may lead to "repeated consumption by Feishu"? Because I don't understand the difference between retrying and "repeated consumption by Feishu".
能否说明一下在重试的时候“会导致飞书重复消费”会出现什么不正确的现象或者逻辑?因为我没明白重试和“飞书重复消费”有什么区别。
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.lark.sink;
+
+import static org.apache.eventmesh.connector.lark.sink.connector.LarkSinkConnector.getTenantAccessToken;
+
+import org.apache.eventmesh.connector.lark.ConnectRecordExtensionKeys;
+import org.apache.eventmesh.connector.lark.config.LarkMessageTemplateType;
+import org.apache.eventmesh.connector.lark.sink.config.SinkConnectorConfig;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+
+import org.apache.commons.text.StringEscapeUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import com.github.rholder.retry.Attempt;
+import com.github.rholder.retry.RetryException;
+import com.github.rholder.retry.RetryListener;
+import com.github.rholder.retry.Retryer;
+import com.github.rholder.retry.RetryerBuilder;
+import com.github.rholder.retry.StopStrategies;
+import com.github.rholder.retry.WaitStrategies;
+import com.lark.oapi.Client;
+import com.lark.oapi.card.enums.MessageCardHeaderTemplateEnum;
+import com.lark.oapi.card.model.MessageCard;
+import com.lark.oapi.card.model.MessageCardConfig;
+import com.lark.oapi.card.model.MessageCardElement;
+import com.lark.oapi.card.model.MessageCardHeader;
+import com.lark.oapi.card.model.MessageCardMarkdown;
+import com.lark.oapi.card.model.MessageCardPlainText;
+import com.lark.oapi.core.httpclient.OkHttpTransport;
+import com.lark.oapi.core.request.RequestOptions;
+import com.lark.oapi.core.utils.Lists;
+import com.lark.oapi.okhttp.OkHttpClient;
+import com.lark.oapi.service.im.v1.ImService;
+import com.lark.oapi.service.im.v1.enums.MsgTypeEnum;
+import com.lark.oapi.service.im.v1.model.CreateMessageReq;
+import com.lark.oapi.service.im.v1.model.CreateMessageReqBody;
+import com.lark.oapi.service.im.v1.model.CreateMessageResp;
+import com.lark.oapi.service.im.v1.model.ext.MessageText;
+
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ImServiceHandler {
+
+ private static final ConcurrentHashMap<ConnectRecord, CreateMessageReq> UN_ACK_REQ = new ConcurrentHashMap<>();
Review Comment:
> The original code will re-inject a new UUID every time it is retransmitted, which will lead to repeated consumption by Feishu.
Could you explain what incorrect phenomenon or logic may occur when retry, which may lead to "repeated consumption by Feishu"? Because I don't understand the difference between retrying and "repeated consumption by Feishu".
能否说明一下在重试的时候“会导致飞书重复消费”会出现什么不正确的现象或者逻辑?因为我没明白重试和“飞书重复消费”有什么区别。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415791656
##########
eventmesh-connectors/eventmesh-connector-lark/src/main/java/org/apache/eventmesh/connector/lark/sink/ImServiceHandler.java:
##########
@@ -93,76 +102,154 @@ public static ImServiceHandler create(SinkConnectorConfig sinkConnectorConfig) {
.build()
.im();
- Map<String, List<String>> headers = new HashMap<>();
- headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
-
- imServiceHandler.requestOptions = RequestOptions.newBuilder()
- .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
- .headers(headers)
- .build();
-
long fixedWait = Long.parseLong(sinkConnectorConfig.getRetryDelayInMills());
int maxRetryTimes = Integer.parseInt(sinkConnectorConfig.getMaxRetryTimes()) + 1;
- imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
- .retryIfException()
- .retryIfResult(Objects::nonNull)
- .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
- .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
- .withRetryListener(new RetryListener() {
- @SneakyThrows
- @Override
- public <V> void onRetry(Attempt<V> attempt) {
- long times = attempt.getAttemptNumber();
- if (times > 1) {
- log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
- }
- if (times == maxRetryTimes) {
- UN_ACK_REQ.remove(attempt.get());
+ if (Boolean.parseBoolean(sinkConnectorConfig.getSinkAsync())) {
+ int availableProcessors = Runtime.getRuntime().availableProcessors();
+ imServiceHandler.sinkAsyncWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-sinkAsyncWorker");
+ return thread;
+ });
+
+ imServiceHandler.cleanerWorker = Executors.newFixedThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-cleanerWorker");
+ return thread;
+ });
+
+ imServiceHandler.retryWorker = Executors.newScheduledThreadPool(availableProcessors, r -> {
+ Thread thread = new Thread(r);
+ thread.setDaemon(true);
+ thread.setName("eventmesh-connector-lark-retryWorker");
+ return thread;
+ });
+ } else {
+ imServiceHandler.retryer = RetryerBuilder.<ConnectRecord>newBuilder()
+ .retryIfException()
+ .retryIfResult(Objects::nonNull)
+ .withWaitStrategy(WaitStrategies.fixedWait(fixedWait, TimeUnit.MILLISECONDS))
+ .withStopStrategy(StopStrategies.stopAfterAttempt(maxRetryTimes))
+ .withRetryListener(new RetryListener() {
+ @SneakyThrows
+ @Override
+ public <V> void onRetry(Attempt<V> attempt) {
+
+ long times = attempt.getAttemptNumber();
+ if (times > 1) {
+ redoSinkNum.increment();
+ log.info("Total redo sink task num : [{}]", redoSinkNum.sum());
+ log.warn("Retry sink event to lark | times=[{}]", attempt.getAttemptNumber() - 1);
+ }
}
- }
- })
- .build();
+ })
+ .build();
+ }
+
return imServiceHandler;
}
public void sink(ConnectRecord connectRecord) throws ExecutionException, RetryException {
- retryer.call(() -> {
- CreateMessageReq createMessageReq = UN_ACK_REQ.computeIfAbsent(connectRecord, (k) -> {
- CreateMessageReqBody.Builder bodyBuilder = CreateMessageReqBody.newBuilder()
- .receiveId(sinkConnectorConfig.getReceiveId())
- .uuid(UUID.randomUUID().toString());
-
- String templateTypeKey = connectRecord.getExtension(ConnectRecordExtensionKeys.TEMPLATE_TYPE_4_LARK);
- if (null == templateTypeKey || "null".equals(templateTypeKey)) {
- templateTypeKey = LarkMessageTemplateType.PLAIN_TEXT.getTemplateKey();
- }
- LarkMessageTemplateType templateType = LarkMessageTemplateType.of(templateTypeKey);
- if (LarkMessageTemplateType.PLAIN_TEXT == templateType) {
- bodyBuilder.content(createTextContent(connectRecord))
- .msgType(MsgTypeEnum.MSG_TYPE_TEXT.getValue());
- } else if (LarkMessageTemplateType.MARKDOWN == templateType) {
- String title = Optional.ofNullable(connectRecord.getExtension(ConnectRecordExtensionKeys.MARKDOWN_MESSAGE_TITLE_4_LARK))
- .orElse("EventMesh-Message");
- bodyBuilder.content(createInteractiveContent(connectRecord, title))
- .msgType(MsgTypeEnum.MSG_TYPE_INTERACTIVE.getValue());
- }
+ Map<String, List<String>> headers = new HashMap<>();
+ headers.put("Content-Type", Lists.newArrayList("application/json; charset=utf-8"));
- return CreateMessageReq.newBuilder()
- .receiveIdType(sinkConnectorConfig.getReceiveIdType())
- .createMessageReqBody(bodyBuilder.build())
- .build();
- });
+ RequestOptions requestOptions = RequestOptions.newBuilder()
+ .tenantAccessToken(getTenantAccessToken(sinkConnectorConfig.getAppId(), sinkConnectorConfig.getAppSecret()))
+ .headers(headers)
+ .build();
+
+ retryer.call(() -> {
+ CreateMessageReq createMessageReq = convertCreateMessageReq(connectRecord);
CreateMessageResp resp = imService.message().create(createMessageReq, requestOptions);
if (resp.getCode() != 0) {
log.warn("Sinking event to lark failure | code:[{}] | msg:[{}] | err:[{}]", resp.getCode(), resp.getMsg(), resp.getError());
return connectRecord;
}
-
- UN_ACK_REQ.remove(connectRecord);
return null;
});
}
+ public void sinkAsync(ConnectRecord connectRecord) {
Review Comment:
I would like to know if there are any hidden dangers in _submitting a task to the thread pool every time a message is sent_?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415738152
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
If this annotation is not added, the following exception message will be thrown
```java
org.mockito.exceptions.misusing.UnnecessaryStubbingException:
Unnecessary stubbings detected.
Clean & maintainable test code requires zero unnecessary code.
Following stubbings are unnecessary (click to navigate to relevant line of code):
1. -> at org.apache.eventmesh.connector.lark.sink.LarkSinkConnectorTest.setup(LarkSinkConnectorTest.java:68)
Please remove unnecessary stubbings or use 'lenient' strictness. More info: javadoc for UnnecessaryStubbingException class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415740183
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
Also needs to be corrected, MockitoSettings defaults to `Strictness.STRICT_STUBS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415755336
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
> Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
What?! Thank you for correcting me!
@yanrongzhen Previously, in one of your PR, I made a mistake https://github.com/apache/eventmesh/pull/4582#discussion_r1405959281 . Sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415755336
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
> Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
What?! Thank you for correcting me!
@yanrongzhen Previously, in one of your PR, I made a mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4598] Improve eventmesh-connector-lark (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4599:
URL: https://github.com/apache/eventmesh/pull/4599#discussion_r1415755336
##########
eventmesh-connectors/eventmesh-connector-lark/src/test/java/org/apache/eventmesh/connector/lark/sink/LarkSinkConnectorTest.java:
##########
@@ -43,12 +43,17 @@
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
Review Comment:
> Another point that needs to be corrected is that MockitoSettings defaults to `Strictness.STRICT_STUBS`.
What?! Thank you for correcting me!
@yanrongzhen Previously, in one of your PR, I made a mistake. Sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org