You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "SteveYurongSu (via GitHub)" <gi...@apache.org> on 2023/01/26 17:55:36 UTC
[PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
SteveYurongSu opened a new pull request, #1166:
URL: https://github.com/apache/streampipes/pull/1166
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<!--
Thanks for contributing! Here are some tips you can follow to help us incorporate your contribution quickly and easily:
1. If this is your first time, please read our contributor guidelines:
- https://streampipes.apache.org/getinvolved.html
- https://cwiki.apache.org/confluence/display/STREAMPIPES/Getting+Started
2. Make sure the PR title is formatted like: `[#<GitHub issue id>] PR title ...`
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., `[WIP][#<GitHub issue id>] PR title ...`.
4. Please write your PR title to summarize what this PR proposes/fixes.
5. Link the PR to the corresponding GitHub issue (if present) in the `Development` section in the right menu bar.
6. Be sure to keep the PR description updated to reflect all changes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
8. Make sure tests pass via `mvn clean install`.
9. (Optional) If the contribution is large, please file an Apache ICLA
- http://apache.org/licenses/icla.pdf
-->
### Purpose
<!--
Please clarify what changes you are proposing and describe how those changes will address the issue.
Furthermore, describe potential consequences the changes might have.
-->
Recently, I was trying to play around streampipes and found that iotdb sink could not work properly.
I checked the source code and found that the current JDBC based implementation of iotdb sink is outdated, especially for the current production versions of iotdb (0.13.x and 1.0.x).
Iād like to contribute a brand new iotdb sink implementation, which has the following features:
- Support production ready versions of IoTDB (version >= 0.13.0)
- Support by iotdb-session API, which can provide higher throughput and lower latency
- Support schema-less write: no need to create storage group (database) before insertion
See #1146 for more detail :D
### Remarks
<!--
Is there anything left we need to pay attention on?
Are there some references that might be important? E.g. links to Confluence, or discussions
on the mailing list or GitHub.
-->
PR introduces (a) breaking change(s): no
PR introduces (a) deprecation(s): yes
Old versions (< v0.13) of IoTDB are no longer supported.
### Tests
I setup a pipeline as the following:
<img width="1427" alt="image" src="https://user-images.githubusercontent.com/30497621/214911642-65693667-1e2e-4331-8536-4bc283070dfb.png">
Then I checked the data that IoTDB received:
<img width="1197" alt="image" src="https://user-images.githubusercontent.com/30497621/214911861-7a8fb705-701d-4226-a321-6a0818cc9cd5.png">
<img width="1197" alt="image" src="https://user-images.githubusercontent.com/30497621/214912001-c0d238cc-16df-40a6-9c7f-ac323b4af2aa.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406476089
Thank you @SteveYurongSu !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406353184
Awsome, thanks a lot.
I am excited to see how this will be implemented by IoTDB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe merged PR #1166:
URL: https://github.com/apache/streampipes/pull/1166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406335001
Hi @SteveYurongSu, thank you so much for this PR and updating the IoTDB implementation.
I have a question regarding the "session".
The session is created in the `onInvocation` and closed in the `onDetach` and what happens if the connection dies during the execution of the pipeline. Would it be possible to add a reconnection in the `onEvent` for this case or is this not necessary?
Maybe this can be done when the "IoTDBConnectionException" occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1405411871
@SteveYurongSu Thank you so much for this awesome contribution šš¼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406419345
Hi @bossenti, here is the update š
In streampipes-extensions/streampipes-sinks-databases-jvm/src/main/java/org/apache/streampipes/sinks/databases/jvm/iotdb/IotDb.java
I implemented the IoTDB sink by `SessionPool` instead of `Session`.
About `SessionPool`:
> We provide a connection pool (`SessionPool) for Native API. Using the interface, you need to define the pool size.
>
> If you can not get a session connection in 60 seconds, there is a warning log but the program will hang.
>
> If a session has finished an operation, it will be put back to the pool automatically. **If a session connection is broken, the session will be removed automatically and the pool will try to create a new session and redo the operation.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406476464
@SteveYurongSu thanks a lot for the super fast update.
I really like the new implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#1146] Session based implementation of IoTDB sink (fix issues in current JDBC based implementation) (streampipes)
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on PR #1166:
URL: https://github.com/apache/streampipes/pull/1166#issuecomment-1406349847
Hi @tenthe, thanks a lot for your reviewing.
IoTDB has a native mechanism to handle the case that the connection dies. I will update the PR with the mechanism ASAP :-)
> Hi @SteveYurongSu, thank you so much for this PR and updating the IoTDB implementation.
> I tried it locally and everything worked fine. Thanks for updating the IoTDB version in the CLI.
>
> I have a question regarding the "session".
> The session is created in the `onInvocation` and closed in the `onDetach` and what happens if the connection dies during the execution of the pipeline. Would it be possible to add a reconnection in the `onEvent` for this case or is this not necessary?
> Maybe this can be done when the "IoTDBConnectionException" occurs.
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org