You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by GitBox <gi...@apache.org> on 2022/08/02 06:22:08 UTC
[GitHub] [incubator-streampipes] tenthe opened a new pull request, #101: Streampipes 563 Refactor Data Lake Sink
tenthe opened a new pull request, #101:
URL: https://github.com/apache/incubator-streampipes/pull/101
<!--
Thanks for contributing! Here are some tips you can follow to help us incorporate your contribution quickly and easily:
1. If this is your first time, please read our contributor guidelines:
- https://streampipes.apache.org/getinvolved.html
- https://cwiki.apache.org/confluence/display/STREAMPIPES/Getting+Started
2. Make sure the PR title is formatted like: `[STREAMPIPES-<Jira issue #>] PR title ...`
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., `[WIP][STREAMPIPES-<Jira issue #>] PR title ...`.
4. Please write your PR title to summarize what this PR proposes/fixes.
5. Be sure to keep the PR description updated to reflect all changes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. Make sure tests pass via `mvn clean install`.
8. (Optional) If the contribution is large, please file an Apache ICLA
- http://apache.org/licenses/icla.pdf
-->
### Purpose
<!--
Please clarify what changes you are proposing and describe how those changes will address the issue.
-->
Refactor Data Lake Sink and code clean up
### Approach
<!--
Describe how you are implementing the solutions along with the design details.
-->
Before, the logic to write data to the data explorer was located in the data explorer sink. In the PR this logic was moved to the package streampipes-data-explorer-commons. Now we have a clean API to write data into the time-series database. This API can also be used by other modules and is not restricted to the internally used data explorer sink.
### Samples
<!--
Provide high-level details about the samples related to this feature.
-->
```java
// initialize in onInvocation method
this.timeSeriesStore = new TimeSeriesStore(
runtimeContext.getConfigStore().getConfig(),
runtimeContext.getStreamPipesClient(),
measure,
true);
...
// store events in onEvent method
this.timeSeriesStore.onEvent(event);
```
### Remarks
<!--
List related issues/PRs, link to discussions in the mailing list, todo items, or any other notes related to the PR.
-->
Fixes: [STREAMPIPES-563](https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-563)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampipes] dominikriemer commented on a diff in pull request #101: Streampipes 563 Refactor Data Lake Sink
Posted by GitBox <gi...@apache.org>.
dominikriemer commented on code in PR #101:
URL: https://github.com/apache/incubator-streampipes/pull/101#discussion_r935174733
##########
streampipes-extensions/streampipes-sinks-internal-jvm/pom.xml:
##########
@@ -56,6 +56,7 @@
</dependency>
<!-- External dependencies -->
+ <!-- TODO remove after refactoring -->
Review Comment:
remove comment
##########
streampipes-platform-services/src/main/java/org/apache/streampipes/ps/DataLakeResourceV4.java:
##########
@@ -43,10 +43,6 @@
import static org.apache.streampipes.dataexplorer.v4.SupportedDataLakeQueryParameters.*;
-class Placeholder {
Review Comment:
I think this is needed for the Swagger API doc generation to work
##########
streampipes-data-explorer/src/main/java/org/apache/streampipes/dataexplorer/DataLakeNoUserManagementV3.java:
##########
@@ -39,7 +42,9 @@ public boolean addDataLake(String measure, EventSchema eventSchema) {
return false;
}
} else {
- getDataLakeStorage().storeDataLakeMeasure(new DataLakeMeasure(measure, eventSchema));
+ DataLakeMeasure dataLakeMeasure = new DataLakeMeasure(measure, eventSchema);
+ dataLakeMeasure.setSchemaVersion(DataLakeMeasure.CURRENT_SCHEMA_VERSION);
Review Comment:
Can the current schema version be set automatically in the constructor for new instances?
##########
streampipes-extensions/streampipes-sinks-internal-jvm/pom.xml:
##########
@@ -64,6 +65,8 @@
<groupId>org.influxdb</groupId>
<artifactId>influxdb-java</artifactId>
</dependency>
+ <!-- TODO remove after refactoring -->
Review Comment:
remove comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampipes] tenthe merged pull request #101: Streampipes 563 Refactor Data Lake Sink
Posted by GitBox <gi...@apache.org>.
tenthe merged PR #101:
URL: https://github.com/apache/incubator-streampipes/pull/101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org