You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eventmesh.apache.org by "Fungx (via GitHub)" <gi...@apache.org> on 2023/12/10 07:09:47 UTC
[PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Fungx opened a new pull request, #4634:
URL: https://github.com/apache/eventmesh/pull/4634
<!--
### Contribution Checklist
- Name the pull request in the form "[ISSUE #XXXX] Title of the pull request",
where *XXXX* should be replaced by the actual issue number.
Skip *[ISSUE #XXXX]* if there is no associated github issue for this pull request.
- Fill out the template below to describe the changes contributed by the pull request.
That will give reviewers the context they need to do the review.
- Each pull request should address only one issue.
Please do not mix up code from multiple issues.
- Each commit in the pull request should have a meaningful commit message.
- Once all items of the checklist are addressed, remove the above text and this checklist,
leaving only the filled out template below.
(The sections below can be removed for hotfixes of typos)
-->
<!--
(If this PR fixes a GitHub issue, please add `Fixes #<XXX>` or `Closes #<XXX>`.)
-->
Fixes #4618.
### Motivation
Add Http source connector.
### Modifications
- Add Http source connector
- Add unit tests for Http source connector
### Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs. Please refer to [http-protocol-binding](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421740428
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
> Actually, I'm confused about `known-dependencies.txt`. Why do we need to manually add dependencies here? Since we are already using a build tool.
I don't fully understand either, I only know that the script check-dependencies.sh for checking dependencies will use this file to check. I also want to know the purpose of checking @xwm1992, and whether there is a better way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1849242557
## [Codecov](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `30 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`ee9ee4c`)](https://app.codecov.io/gh/apache/eventmesh/commit/ee9ee4c63670b57b832898624a061335153bb67c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 16.93% compared to head [(`115e599`)](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 17.37%.
> Report is 2 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...tor/http/source/connector/HttpSourceConnector.java](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2h0dHAvc291cmNlL2Nvbm5lY3Rvci9IdHRwU291cmNlQ29ubmVjdG9yLmphdmE=) | 71.42% | [13 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...tmesh/connector/http/server/HttpConnectServer.java](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2h0dHAvc2VydmVyL0h0dHBDb25uZWN0U2VydmVyLmphdmE=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ntmesh/connector/http/config/HttpServerConfig.java](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2h0dHAvY29uZmlnL0h0dHBTZXJ2ZXJDb25maWcuamF2YQ==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ctor/http/source/config/SourceConnectorConfig.java](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2h0dHAvc291cmNlL2NvbmZpZy9Tb3VyY2VDb25uZWN0b3JDb25maWcuamF2YQ==) | 60.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...connector/http/source/config/HttpSourceConfig.java](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXZlbnRtZXNoLWNvbm5lY3RvcnMvZXZlbnRtZXNoLWNvbm5lY3Rvci1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9ldmVudG1lc2gvY29ubmVjdG9yL2h0dHAvc291cmNlL2NvbmZpZy9IdHRwU291cmNlQ29uZmlnLmphdmE=) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4634 +/- ##
============================================
+ Coverage 16.93% 17.37% +0.44%
- Complexity 1708 1757 +49
============================================
Files 790 797 +7
Lines 29526 29767 +241
Branches 2550 2571 +21
============================================
+ Hits 5001 5173 +172
- Misses 24061 24114 +53
- Partials 464 480 +16
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/eventmesh/pull/4634?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421740428
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
> Actually, I'm confused about `known-dependencies.txt`. Why do we need to manually add dependencies here? Since we are already using a build tool.
I don't fully understand either, I only know that the script check-dependencies.sh for checking dependencies will use this file to check. I also want to know the purpose of checking @xwm1992, and whether there is a better way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1423516228
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
I have manually comfirmed that these newly added dependencies(vertx , netty & cloudevents-http-vertx) are under the ASF 3rd party license policy, since they are all using Apache License 2.0. Is this correct?
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
I have manually comfirmed that these newly added dependencies(vertx , netty & cloudevents-http-vertx) are under the ASF 3rd party license policy, since they are all using Apache License 2.0. Is this correct? @xwm1992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421740428
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
> Actually, I'm confused about `known-dependencies.txt`. Why do we need to manually add dependencies here? Since we are already using a build tool.
I don't fully understand either, I only know that the script check-dependencies.sh for checking dependencies will use this file to check. I also want to know the purpose of checking and whether there is a better way. @xwm1992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1869942423
> > Would you be willing to write a document([#4601 (comment)](https://github.com/apache/eventmesh/issues/4601#issuecomment-1838736726)) for this connector? If you are willing, they can be written in this PR or in a new PR.
>
> Of course. I'll write it in a new PR.
@Fungx In the future, community will mainly maintain documents in [eventmesh-site](https://github.com/apache/eventmesh-site). Therefore, if it is convenient for you, please write the new English documents in this [directory](https://github.com/apache/eventmesh-site/tree/master/docs/design-document/03-connect) of eventmesh-site, and Chinese documents in the [directory](https://github.com/apache/eventmesh-site/tree/master/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1870840598
> > > Would you be willing to write a document([#4601 (comment)](https://github.com/apache/eventmesh/issues/4601#issuecomment-1838736726)) for this connector? If you are willing, they can be written in this PR or in a new PR.
> >
> >
> > Of course. I'll write it in a new PR.
>
> @Fungx In the future, community will mainly maintain documents in [eventmesh-site](https://github.com/apache/eventmesh-site). Therefore, if it is convenient for you, please write the new English documents in this [directory](https://github.com/apache/eventmesh-site/tree/master/docs/design-document/03-connect) of eventmesh-site, and Chinese documents in the [directory](https://github.com/apache/eventmesh-site/tree/master/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect).
Sure. Leave it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1848933220
> Would you be willing to write a document([#4601 (comment)](https://github.com/apache/eventmesh/issues/4601#issuecomment-1838736726)) for this connector? If you are willing, they can be written in this PR or in a new PR.
Of course. I'll write it in a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 merged PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: dev-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1422494209
##########
eventmesh-connectors/eventmesh-connector-http/src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.http.source.connector;
+
+import org.apache.eventmesh.common.exception.EventMeshException;
+import org.apache.eventmesh.connector.http.source.config.HttpSourceConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SourceConnectorContext;
+import org.apache.eventmesh.openconnect.api.source.Source;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+import org.apache.eventmesh.openconnect.util.CloudEventUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import io.cloudevents.CloudEvent;
+import io.cloudevents.core.message.MessageReader;
+import io.cloudevents.http.vertx.VertxMessageFactory;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.vertx.core.Vertx;
+import io.vertx.core.http.HttpMethod;
+import io.vertx.core.http.HttpServer;
+import io.vertx.core.http.HttpServerOptions;
+import io.vertx.ext.web.Router;
+import io.vertx.ext.web.handler.LoggerHandler;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class HttpSourceConnector implements Source {
+
+ private static final int DEFAULT_BATCH_SIZE = 10;
+
+ private HttpSourceConfig sourceConfig;
+ private BlockingQueue<CloudEvent> queue;
+ private HttpServer server;
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return HttpSourceConfig.class;
+ }
+
+ @Override
+ public void init(Config config) throws Exception {
+ this.sourceConfig = (HttpSourceConfig) config;
+ doInit();
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) throws Exception {
+ SourceConnectorContext sourceConnectorContext = (SourceConnectorContext) connectorContext;
+ this.sourceConfig = (HttpSourceConfig) sourceConnectorContext.getSourceConfig();
+ doInit();
+ }
+
+ private void doInit() throws Exception {
+ this.queue = new LinkedBlockingQueue<>(1000);
Review Comment:
Fixed. And now the trace stack will be logged by the outside `SourceWorker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1422037387
##########
eventmesh-connectors/eventmesh-connector-http/src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.http.source.connector;
+
+import org.apache.eventmesh.common.exception.EventMeshException;
+import org.apache.eventmesh.connector.http.source.config.HttpSourceConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SourceConnectorContext;
+import org.apache.eventmesh.openconnect.api.source.Source;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+import org.apache.eventmesh.openconnect.util.CloudEventUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import io.cloudevents.CloudEvent;
+import io.cloudevents.core.message.MessageReader;
+import io.cloudevents.http.vertx.VertxMessageFactory;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.vertx.core.Vertx;
+import io.vertx.core.http.HttpMethod;
+import io.vertx.core.http.HttpServer;
+import io.vertx.core.http.HttpServerOptions;
+import io.vertx.ext.web.Router;
+import io.vertx.ext.web.handler.LoggerHandler;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class HttpSourceConnector implements Source {
+
+ private static final int DEFAULT_BATCH_SIZE = 10;
+
+ private HttpSourceConfig sourceConfig;
+ private BlockingQueue<CloudEvent> queue;
+ private HttpServer server;
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return HttpSourceConfig.class;
+ }
+
+ @Override
+ public void init(Config config) throws Exception {
+ this.sourceConfig = (HttpSourceConfig) config;
+ doInit();
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) throws Exception {
+ SourceConnectorContext sourceConnectorContext = (SourceConnectorContext) connectorContext;
+ this.sourceConfig = (HttpSourceConfig) sourceConnectorContext.getSourceConfig();
+ doInit();
+ }
+
+ private void doInit() throws Exception {
+ this.queue = new LinkedBlockingQueue<>(1000);
+
+ final Vertx vertx = Vertx.vertx();
+ final Router router = Router.router(vertx);
+ router.route()
+ .path(this.sourceConfig.connectorConfig.getPath())
+ .method(HttpMethod.POST)
+ .handler(LoggerHandler.create())
+ .handler(ctx -> {
+ VertxMessageFactory.createReader(ctx.request())
+ .map(MessageReader::toEvent)
Review Comment:
> Is there a custom conversion method available here to directly convert data into `ConnectorRecord`? This will reduce one data conversion in `poll()`. If possible, you can optimize in this PR or in a new PR.
In order to be compliant with [cloudevent-spec](https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md), converting an http request into `connectRecord` may introduce a lot of http parsing code duplicated in cloudevent-sdk.
For future maintenance, I'm wondering if this is worth it or if there is a better way, and if so, maybe we should implement it in a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "hhuang1231 (via GitHub)" <gi...@apache.org>.
hhuang1231 commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1849661672
Maybe this question shouldn't be asked here, but I still want to know why most sourceConnectors don't implement the `commit()` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421742628
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
> Actually, I'm confused about `known-dependencies.txt`. Why do we need to manually add dependencies here? Since we are already using a build tool.
I don't fully understand either, I only know that the script check-dependencies.sh for checking dependencies will use this file to check. I also want to know the purpose of using this method of checking @ruanwenjun , and whether there is a better way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Pil0tXia (via GitHub)" <gi...@apache.org>.
Pil0tXia commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1422330375
##########
eventmesh-connectors/eventmesh-connector-http/src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.http.source.connector;
+
+import org.apache.eventmesh.common.exception.EventMeshException;
+import org.apache.eventmesh.connector.http.source.config.HttpSourceConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SourceConnectorContext;
+import org.apache.eventmesh.openconnect.api.source.Source;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+import org.apache.eventmesh.openconnect.util.CloudEventUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import io.cloudevents.CloudEvent;
+import io.cloudevents.core.message.MessageReader;
+import io.cloudevents.http.vertx.VertxMessageFactory;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.vertx.core.Vertx;
+import io.vertx.core.http.HttpMethod;
+import io.vertx.core.http.HttpServer;
+import io.vertx.core.http.HttpServerOptions;
+import io.vertx.ext.web.Router;
+import io.vertx.ext.web.handler.LoggerHandler;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class HttpSourceConnector implements Source {
+
+ private static final int DEFAULT_BATCH_SIZE = 10;
+
+ private HttpSourceConfig sourceConfig;
+ private BlockingQueue<CloudEvent> queue;
+ private HttpServer server;
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return HttpSourceConfig.class;
+ }
+
+ @Override
+ public void init(Config config) throws Exception {
+ this.sourceConfig = (HttpSourceConfig) config;
+ doInit();
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) throws Exception {
+ SourceConnectorContext sourceConnectorContext = (SourceConnectorContext) connectorContext;
+ this.sourceConfig = (HttpSourceConfig) sourceConnectorContext.getSourceConfig();
+ doInit();
+ }
+
+ private void doInit() throws Exception {
+ this.queue = new LinkedBlockingQueue<>(1000);
Review Comment:
Redundant `throws Exception` at `doInit()`, `start()` and `stop()` can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "Fungx (via GitHub)" <gi...@apache.org>.
Fungx commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421736432
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
Many newly added dependencies are indirectly imported by vertx. I can't pass the `License Check` of CI/CD after removing them.
Actually, I'm confused about `known-dependencies.txt`. Why do we need to manually add dependencies here? Since we are already using a build tool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1421699370
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
Apart from cloudevents-http-vertx and vertx-web, are there any unnecessary dependencies in the newly added dependencies?
##########
eventmesh-connectors/eventmesh-connector-http/src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.http.source.connector;
+
+import org.apache.eventmesh.common.exception.EventMeshException;
+import org.apache.eventmesh.connector.http.source.config.HttpSourceConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SourceConnectorContext;
+import org.apache.eventmesh.openconnect.api.source.Source;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+import org.apache.eventmesh.openconnect.util.CloudEventUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import io.cloudevents.CloudEvent;
+import io.cloudevents.core.message.MessageReader;
+import io.cloudevents.http.vertx.VertxMessageFactory;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.vertx.core.Vertx;
+import io.vertx.core.http.HttpMethod;
+import io.vertx.core.http.HttpServer;
+import io.vertx.core.http.HttpServerOptions;
+import io.vertx.ext.web.Router;
+import io.vertx.ext.web.handler.LoggerHandler;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class HttpSourceConnector implements Source {
+
+ private static final int DEFAULT_BATCH_SIZE = 10;
+
+ private HttpSourceConfig sourceConfig;
+ private BlockingQueue<CloudEvent> queue;
+ private HttpServer server;
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return HttpSourceConfig.class;
+ }
+
+ @Override
+ public void init(Config config) throws Exception {
+ this.sourceConfig = (HttpSourceConfig) config;
+ doInit();
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) throws Exception {
+ SourceConnectorContext sourceConnectorContext = (SourceConnectorContext) connectorContext;
+ this.sourceConfig = (HttpSourceConfig) sourceConnectorContext.getSourceConfig();
+ doInit();
+ }
+
+ private void doInit() throws Exception {
+ this.queue = new LinkedBlockingQueue<>(1000);
+
+ final Vertx vertx = Vertx.vertx();
+ final Router router = Router.router(vertx);
+ router.route()
+ .path(this.sourceConfig.connectorConfig.getPath())
+ .method(HttpMethod.POST)
+ .handler(LoggerHandler.create())
+ .handler(ctx -> {
+ VertxMessageFactory.createReader(ctx.request())
+ .map(MessageReader::toEvent)
Review Comment:
Is there a custom conversion method available here to directly convert data into `ConnectorRecord`? This will reduce one data conversion in `poll()`. If possible, you can optimize in this PR or in a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1422176874
##########
eventmesh-connectors/eventmesh-connector-http/src/main/java/org/apache/eventmesh/connector/http/source/connector/HttpSourceConnector.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eventmesh.connector.http.source.connector;
+
+import org.apache.eventmesh.common.exception.EventMeshException;
+import org.apache.eventmesh.connector.http.source.config.HttpSourceConfig;
+import org.apache.eventmesh.openconnect.api.config.Config;
+import org.apache.eventmesh.openconnect.api.connector.ConnectorContext;
+import org.apache.eventmesh.openconnect.api.connector.SourceConnectorContext;
+import org.apache.eventmesh.openconnect.api.source.Source;
+import org.apache.eventmesh.openconnect.offsetmgmt.api.data.ConnectRecord;
+import org.apache.eventmesh.openconnect.util.CloudEventUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import io.cloudevents.CloudEvent;
+import io.cloudevents.core.message.MessageReader;
+import io.cloudevents.http.vertx.VertxMessageFactory;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.vertx.core.Vertx;
+import io.vertx.core.http.HttpMethod;
+import io.vertx.core.http.HttpServer;
+import io.vertx.core.http.HttpServerOptions;
+import io.vertx.ext.web.Router;
+import io.vertx.ext.web.handler.LoggerHandler;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class HttpSourceConnector implements Source {
+
+ private static final int DEFAULT_BATCH_SIZE = 10;
+
+ private HttpSourceConfig sourceConfig;
+ private BlockingQueue<CloudEvent> queue;
+ private HttpServer server;
+
+ @Override
+ public Class<? extends Config> configClass() {
+ return HttpSourceConfig.class;
+ }
+
+ @Override
+ public void init(Config config) throws Exception {
+ this.sourceConfig = (HttpSourceConfig) config;
+ doInit();
+ }
+
+ @Override
+ public void init(ConnectorContext connectorContext) throws Exception {
+ SourceConnectorContext sourceConnectorContext = (SourceConnectorContext) connectorContext;
+ this.sourceConfig = (HttpSourceConfig) sourceConnectorContext.getSourceConfig();
+ doInit();
+ }
+
+ private void doInit() throws Exception {
+ this.queue = new LinkedBlockingQueue<>(1000);
+
+ final Vertx vertx = Vertx.vertx();
+ final Router router = Router.router(vertx);
+ router.route()
+ .path(this.sourceConfig.connectorConfig.getPath())
+ .method(HttpMethod.POST)
+ .handler(LoggerHandler.create())
+ .handler(ctx -> {
+ VertxMessageFactory.createReader(ctx.request())
+ .map(MessageReader::toEvent)
Review Comment:
I think it will be worthy. Currently, I don't know if there is a better way, and look forward to your future research or community work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "pandaapo (via GitHub)" <gi...@apache.org>.
pandaapo commented on PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#issuecomment-1849705152
> Maybe this question shouldn't be asked here, but I still want to know why most sourceConnectors don't implement the `commit()` method.
@hhuang1231
`commit()` is mainly used to record the offset of messages that need to be persisted, and to avoid duplicate transmission of such messages as much as possible. However, not all connectors need to implement it, such as this Connector and the Lark Connector you refactored. Connectors such as RocketMQ, RabbitMQ, Kafka, MySQL, etc. should implement to make them more reliable. Of course, there are currently some connectors that need to implement it but have not implemented. The community wants to gradually improve the functionality of a Connector and does not require contributors to implement all functions in one PR.
`commit()`是主要用于记录需要持久化的消息的位移，尽量避免这类消息的重复传输，但不是所有的Connector都需要实现，像该Connector、您重构的Lark Connector等不需要实现。像RocketMQ、RabbitMQ、Kafka、MySQL等这类的Connector，应该实现，这样Connector会更加可靠。当然，目前有一些需要实现的Connector并没有实现，社区是想逐步完善一个Connector的功能，不强制要求贡献这在一次PR中全部实现。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [PR] [ISSUE #4618] Add HTTP source connector (eventmesh)
Posted by "xwm1992 (via GitHub)" <gi...@apache.org>.
xwm1992 commented on code in PR #4634:
URL: https://github.com/apache/eventmesh/pull/4634#discussion_r1423335624
##########
tools/dependency-check/known-dependencies.txt:
##########
@@ -25,6 +25,7 @@ cache-api-1.1.1.jar
checker-qual-3.12.0.jar
cloudevents-api-2.4.2.jar
cloudevents-core-2.4.2.jar
+cloudevents-http-vertx-2.3.0.jar
Review Comment:
We need to use the known-dependencies to recognize which packages are new dependencies and make sure these new dependencies must under the ASF 3rd party license policy. @Fungx @pandaapo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org