You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "SteveYurongSu (via GitHub)" <gi...@apache.org> on 2023/01/31 04:17:56 UTC

[PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

SteveYurongSu opened a new pull request, #1191:
URL: https://github.com/apache/streampipes/pull/1191

   <!--
     ~ Licensed to the Apache Software Foundation (ASF) under one or more
     ~ contributor license agreements.  See the NOTICE file distributed with
     ~ this work for additional information regarding copyright ownership.
     ~ The ASF licenses this file to You under the Apache License, Version 2.0
     ~ (the "License"); you may not use this file except in compliance with
     ~ the License.  You may obtain a copy of the License at
     ~
     ~    http://www.apache.org/licenses/LICENSE-2.0
     ~
     ~ Unless required by applicable law or agreed to in writing, software
     ~ distributed under the License is distributed on an "AS IS" BASIS,
     ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     ~ See the License for the specific language governing permissions and
     ~ limitations under the License.
     ~
     -->
     
     <!--
   Thanks for contributing! Here are some tips you can follow to help us incorporate your contribution quickly and easily:
   1. If this is your first time, please read our contributor guidelines:
       - https://streampipes.apache.org/getinvolved.html
       - https://cwiki.apache.org/confluence/display/STREAMPIPES/Getting+Started
   2. Make sure the PR title is formatted like: `[#<GitHub issue id>] PR title ...`
   3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., `[WIP][#<GitHub issue id>] PR title ...`.
   4. Please write your PR title to summarize what this PR proposes/fixes.
   5. Link the PR to the corresponding GitHub issue (if present) in the `Development` section in the right menu bar. 
   6. Be sure to keep the PR description updated to reflect all changes.
   7. If possible, provide a concise example to reproduce the issue for a faster review.
   8. Make sure tests pass via `mvn clean install`.
   9. (Optional) If the contribution is large, please file an Apache ICLA
       - http://apache.org/licenses/icla.pdf
   -->
   
   ### Background
   [Apache TubeMQ](https://inlong.apache.org/docs/modules/tubemq/overview) (now is a submodule of Apache Inlong) is a trillion-records-scale distributed messaging queue (MQ) system, focuses on data transmission and storage under massive data. Compared to many open source MQ projects, TubeMQ has unique advantages in terms of stability, performance, and low cost.
   
   It offers a variety of features:
   
   - Pluggable transport protocols, such as TCP, SSL
   - Support big-data and streaming ecosystem integration
   - Message retroactivity by time or offset
   - Efficient pull and push consumption model
   - Flexible distributed scale-out deployment architecture
   - Feature-rich administrative dashboard for configuration
   - Authentication and authorization
   
   For more information:
   
   - [TubeMQ VS Kafka](https://inlong.apache.org/docs/modules/tubemq/tubemq_perf_test_vs_Kafka)
   
   ### Approach
   **Connect Adapter:**
   Introduce the TubeMQProtocol for reading data from TubeMQ brokers.
   
   **Sink:**
   Introduce the TubeMQPublisherSink for writing data to TubeMQ brokers.
   
   
   ### Remarks
   <!--
   Is there anything left we need to pay attention on?
   Are there some references that might be important? E.g. links to Confluence, or discussions
   on the mailing list or GitHub.
   -->
   PR introduces (a) breaking change(s): <yes/no>
   no
   
   PR introduces (a) deprecation(s): <yes/no>
   no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#issuecomment-1413214439

   Glad to hear! 
   That would be awesome 🤩 
   We really appreciate our contributions ❤️ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#discussion_r1092251187


##########
installer/cli/deploy/standalone/tubemq/docker-compose.dev.yml:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.4"
+services:
+  iotdb:

Review Comment:
   The service name should be changed to `tubemq`



##########
installer/cli/deploy/standalone/tubemq/docker-compose.yml:
##########
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.4"
+services:
+  kafka:

Review Comment:
   Same as above. Change name to `tubemq`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#discussion_r1093580549


##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/resources/org.apache.streampipes.connect.iiot.protocol.stream.tubemq/documentation.md:
##########
@@ -0,0 +1,32 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+## Apache TubeMQ (InLong)
+
+<p align="center"> 
+    <img src="icon.png" width="150px;" class="pe-image-documentation"/>
+</p>
+
+***
+
+## Description
+
+Consumes messages from an Apache TubeMQ broker.
+
+***

Review Comment:
   Not directly required, but it would be awesome if you could add a short descriptionof the input parameters here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#discussion_r1094089226


##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/resources/org.apache.streampipes.connect.iiot.protocol.stream.tubemq/documentation.md:
##########
@@ -0,0 +1,32 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+## Apache TubeMQ (InLong)
+
+<p align="center"> 
+    <img src="icon.png" width="150px;" class="pe-image-documentation"/>
+</p>
+
+***
+
+## Description
+
+Consumes messages from an Apache TubeMQ broker.
+
+***

Review Comment:
   Thanks a lot 🙏🏼 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#discussion_r1093953798


##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/resources/org.apache.streampipes.connect.iiot.protocol.stream.tubemq/documentation.md:
##########
@@ -0,0 +1,32 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+## Apache TubeMQ (InLong)
+
+<p align="center"> 
+    <img src="icon.png" width="150px;" class="pe-image-documentation"/>
+</p>
+
+***
+
+## Description
+
+Consumes messages from an Apache TubeMQ broker.
+
+***

Review Comment:
   Good point! Description is added :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#issuecomment-1413079105

   > How was your experience with the extensibility of StreamPipes for adapters and sinks? Did you get the required information easily?
   
   Overall, the experience was very good. 
   
   I mainly got the necessary information from the following two links:
   
   - https://streampipes.apache.org/docs/docs/extend-setup.html
   - https://cwiki.apache.org/confluence/display/STREAMPIPES/Extensions
   
   In addition, I have to say that streampipes are too suitable for iot developers! I'm going to contribute some more code related to the IoTDB eco :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on PR #1191:
URL: https://github.com/apache/streampipes/pull/1191#issuecomment-1411574391

   Thanks a lot for the reviewing!
   
   Sorry for the wrong service name. I have fixed them.
   
   > One more hint. For sinks we have a newer API `StreamPipesDataSink`, this requires only one class instead of three. Here are two example [1], [2].
   
   Yes, this time I already implemented `TubeMQPublisherSink` based on the latest API `StreamPipesDataSink`. The implementation experience was much better :)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [#1190] Apache TubeMQ (InLong) Adapter & Sink (streampipes)

Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti merged PR #1191:
URL: https://github.com/apache/streampipes/pull/1191


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org