You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by wi...@apache.org on 2020/04/09 10:33:19 UTC
[incubator-streampipes] branch edge-extensions updated (b0a0708 ->
0da1709)
This is an automated email from the ASF dual-hosted git repository.
wiener pushed a change to branch edge-extensions
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git.
from b0a0708 Merge pull request #13 from EbiDa/timeseries-labeling-tool
new 7eef4f8 move package to org.apache.streampipes, initial node controller
new 82b2c7b Merge remote-tracking branch 'upstream/dev' into dev
new b41493f Add NodeInfo model
new 4ff05c7 Add backend features to fetch registered nodes
new 43bef21 Add initial version of edge device registration
new 723a270 Modify consul config allow configuration of multiple pipeline elements per appId
new 43452ef update node model, add endpoint for node description
new ac3aa17 add nodeName to NodeInfoBuilder
new 85b966a remove indent
new d053778 deploy container via node controller
new bc21e73 Modify Consul registration
new e612a0d Merge branch 'dev' of github.com:fogsy-io/incubator-streampipes into dev
new 04b5e49 Add deployment settings to start pipeline dialog
new 4bef12d Merge branch 'dev' of github.com:fogsy-io/incubator-streampipes into dev
new e18c67f Merge branch 'dev' of https://github.com/fogsy-io/incubator-streampipes into dev
new 6ccfbe7 add method to delete keys in consul
new e05efd8 add rdf property to PipelineElementDockerContainer class
new 0c9c74b add rest endpoint and methods to interact with docker daemon
new 9512c66 add env variable for node_info.yml location
new 0d720c6 add license header
new df99b4d Merge remote-tracking branch 'upstream/dev' into dev
new c20d6af Merge remote-tracking branch 'upstream/dev' into dev
new ed7ccb8 remove not needed import
new 2790fe1 Merge remote-tracking branch 'upstream/dev' into dev
new 58a54ed Modify pipeline element endpoints based on selected node id
new d23bad1 Merge branch 'dev' of github.com:fogsy-io/incubator-streampipes into dev
new 91b9e45 Merge branch 'dev' of https://github.com/fogsy-io/incubator-streampipes into dev
new 02ac39b refactored code to get static node information on start
new 2474af1 Merge remote-tracking branch 'upstream/dev' into dev
new 9505ddc add support for appId as env variable
new 36cd5c3 Modify node info model in UI
new 1ac8be8 Add processors to deployment selection
new e1c5367 add support for pe invokation on different nodes
new 128c866 Merge remote-tracking branch 'upstream/dev' into dev
new b448a78 Merge remote-tracking branch 'upstream/dev' into dev
new 7c305e7 merged latest upstream
new e52cd27 Merge remote-tracking branch 'upstream/dev' into dev
new a75ca63 Merge remote-tracking branch 'upstream/dev' into dev
new 90ae93d refactored ContainerOrchestrator
new 0df98bb refactored node controller
new 34089c9 Merge remote-tracking branch 'upstream/dev' into dev
new 811309a get correct configItems for new pe structure in cosul
new aad63cf add multi-stage docker build for ui
new 6df623a fixed typo
new 20aab17 Merge remote-tracking branch 'upstream/dev' into dev
new 4352e62 differentiate between primary/secondary pipeline elements, visualize in configurations in ui
new 8f96f60 Merge remote-tracking branch 'upstream/dev' into dev
new fa18289 remove multi-stage build
new 1124627 Merge remote-tracking branch 'upstream/dev' into dev
new b1b62eb initial work on jms only grounding
new 620306c Merge remote-tracking branch 'upstream/dev' into dev
new 68ae690 Merge branch 'dev' into jms
new a28a9ad add some comments
new e684f5b Add feature to select protocol based on user preferences
new 21347ad Merge remote-tracking branch 'upstream/dev' into dev
new a4d1229 Merge branch 'dev' into jms
new 46f0735 Merge remote-tracking branch 'upstream/dev' into jms
new 54659b9 add node controller to workflow
new 71ed65b added some performance measurements for pipeline invocation/detach
new cba6a1f Merge remote-tracking branch 'upstream/dev' into dev
new b00f62f skip erforcer plugin in ci build
new 9cdb5de set fogsyio repo in build
new 8bcad7c add qemu* to streampipes-node-controller-container in ci build
new 6bcd716 edit build.yml
new 656cc71 remove dockerginore to prevent ui build to fail, refactored supported app id in node controller
new 0992368 Merge branch 'dev' into jms
new df4115c Merge remote-tracking branch 'upstream/dev' into dev
new 76f41c1 Merge branch 'dev' into jms
new fb27b80 update branch
new f3c543a add gpu resource to node configuration in ui
new 215d13c add horizontal line to node config
new cec8473 new layout to node configuration, some fixes to node env variable for accessible sa resources
new c24cd53 add raw resource metrics incl. memory usage, remove linebreak in node configuration
new a2815ae add /dev/root for rpi filesystem
new 7f439ca shorten OS info for raspbian
new b2fb3aa add /dev/mmcblk0p1 for jetson nano filesystem
new 689c0e1 Merge remote-tracking branch 'upstream/dev' into dev
new 2bf72ad Merge branch 'dev' into jms
new ec6172e Add MQTT protocol support
new ddca71c check if brokerhostname contains scheme information
new e59098b check if brokerhostname contains scheme information
new fd450cc Merge branch 'jms' of github.com:fogsy-io/incubator-streampipes into jms
new e70c293 Assembly JMS hostname in streampipes-messaging-jms module
new 7ef9535 add mqtt to prioritized message settings, remove tcp schema from ActiveMQPublisher
new 7533624 add total memory usage to resource status
new 1ce6d6f refactored node info and node controller config
new 25fde11 change workflow to original state, add node-controller container deployment
new b8d2850 Merge remote-tracking branch 'upstream/dev' into dev
new 0da1709 Merge pull request #15 from fogsy-io/dev
The 5111 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build.yml | 21 +-
pom.xml | 7 +
.../backend/StreamPipesResourceConfig.java | 4 +-
.../streampipes/config/backend/BackendConfig.java | 12 +
.../config/backend/BackendConfigKeys.java | 2 +
.../config/backend/MessagingSettings.java | 16 +-
.../backend/{SpDataFormat.java => SpProtocol.java} | 22 +-
.../streampipes/config/consul/ConsulSpConfig.java | 26 +-
.../development/env | 3 +-
.../master/init/AdapterMasterContainer.java | 2 +-
.../master/management/AdapterMasterManagement.java | 3 +-
.../master/AdapterMasterManagementTest.java | 0
.../master/DescriptionManagementTest.java | 0
.../AdapterTemplateMasterManagementTest.java | 0
.../master/management/SourcesManagementTest.java | 0
.../management/UnitMasterManagementTest.java | 0
.../master/management/WorkerRestClientTest.java | 0
streampipes-connect-container/development/env | 3 +-
streampipes-connect/pom.xml | 10 +
.../streampipes/connect/adapter/Adapter.java | 46 ++-
.../connect/adapter/GroundingService.java | 82 +++++-
.../model/pipeline/AdapterPipelineElement.java | 2 +-
.../elements/SendToBrokerAdapterSink.java | 105 +++++++
...elineElement.java => SendToJmsAdapterSink.java} | 23 +-
.../elements/SendToKafkaAdapterSink.java | 53 +---
...lineElement.java => SendToMqttAdapterSink.java} | 23 +-
.../connect/adapter/GroundingServiceTest.java | 35 +--
.../standalone/init/StandaloneModelSubmitter.java | 10 +
.../apache/streampipes/container/api/Element.java | 14 +-
.../container/api/InvocableElement.java | 5 +-
.../streampipes/container/api/SecElement.java | 8 +-
.../streampipes/container/api/SepaElement.java | 8 +-
.../container/declarer/DataStreamDeclarer.java | 1 +
.../container/init/DeclarersSingleton.java | 17 ++
.../consul/ConsulServiceRegistrationBody.java | 90 ++++++
.../model/consul/HealthCheckConfiguration.java | 49 ++--
.../streampipes/container/util/ConsulUtil.java | 141 +++++++--
.../streampipes/container/util/NodeUtil.java | 120 ++++++++
.../messaging/jms/ActiveMQConsumer.java | 3 +-
.../messaging/jms/ActiveMQPublisher.java | 11 +-
.../streampipes/messaging/jms/ActiveMQUtils.java | 13 +-
streampipes-messaging-mqtt/pom.xml | 4 +
.../streampipes/messaging/mqtt/MqttConsumer.java | 32 +-
.../model/base/ConsumableStreamPipesEntity.java | 35 +++
.../model/base/InvocableStreamPipesEntity.java | 102 +++++++
.../model/connect/adapter/AdapterDescription.java | 14 +-
.../model/graph/DataProcessorInvocation.java | 13 +-
.../model/graph/DataSinkInvocation.java | 13 +-
.../model/grounding/JmsTransportProtocol.java | 5 +
.../org/apache/streampipes/model/node/Node.java | 20 +-
.../streampipes/model/node/NodeBrokerInfo.java | 51 ++--
.../apache/streampipes/model/node/NodeInfo.java | 82 ++++++
.../streampipes/model/node/NodeInfoBuilder.java | 109 +++++++
.../streampipes/model/node/NodeMetadata.java | 60 ++++
.../streampipes/model/node/NodeResources.java | 56 ++++
.../model/node/PipelineElementDockerContainer.java | 138 +++++++++
.../model/node/resources/hardware/CPU.java | 28 +-
.../model/node/resources/hardware/DISK.java | 20 +-
.../resources/hardware/GPU.java} | 37 ++-
.../resources/hardware/HardwareResource.java} | 42 +--
.../model/node/resources/hardware/MEM.java | 21 +-
.../AccessibleSensorActuatorResource.java | 49 ++--
.../model/node/resources/software/Cuda.java | 65 ++---
.../model/node/resources/software/Docker.java | 57 ++++
.../node/resources/software/SoftwareResource.java | 50 ++--
.../org/apache/streampipes/model/util/Cloner.java | 3 +
.../Dockerfile | 13 +-
.../aarch64.Dockerfile | 12 +-
.../arm.Dockerfile | 12 +-
.../development/env | 34 +++
.../pom.xml | 50 ++--
.../container/api/NodeControllerResource.java | 101 +++++++
.../container/api/NodeInfoStatusResource.java | 43 ++-
.../controller/container/api/NodeResource.java | 23 +-
.../controller/container/config/ConfigKeys.java | 34 +++
.../container/config/NodeControllerConfig.java | 218 ++++++++++++++
.../container/init/NodeControllerContainer.java | 73 +++++
.../NodeControllerContainerResourceConfig.java | 25 +-
.../container/ContainerOrchestrator.java | 10 +-
.../management/container/ContainerStatus.java | 12 +-
.../container/management/container/DockerInfo.java | 116 ++++++++
.../container/DockerOrchestratorManager.java | 149 ++++++++++
.../management/container/DockerUtils.java | 323 +++++++++++++++++++++
.../container/management/info/NodeInfoStorage.java | 240 +++++++++++++++
.../management/node/NodeJanitorManager.java | 63 ++++
.../management/pe/PipelineElementManager.java | 25 +-
.../management/resource/ResourceManager.java | 205 +++++++++++++
.../src/main/resources/banner.txt | 2 +-
.../execution/http/DataSetEntityUrlGenerator.java | 25 +-
.../http/EndpointUrlGenerator.java} | 20 +-
.../manager/execution/http/GraphSubmitter.java | 82 ++++--
.../manager/execution/http/HttpRequestBuilder.java | 37 ++-
.../http/InvocableEntityUrlGenerator.java | 106 +++++++
.../manager/matching/InvocationGraphBuilder.java | 1 +
.../manager/matching/ProtocolSelector.java | 37 ++-
.../manager/node/AvailableNodesFetcher.java | 65 +++++
.../streampipes/manager/operations/Operations.java | 6 +
.../rest/api/{ICouchdb.java => INode.java} | 6 +-
.../apache/streampipes/rest/impl/ConsulConfig.java | 31 +-
.../rest/impl/{DataStream.java => Node.java} | 24 +-
.../rest/impl/PipelineWithUserResource.java | 42 ++-
.../AbstractNotificationSubscriber.java | 2 +-
.../streampipes/rest/impl/ConsulConfigTest.java | 2 +-
.../apache/streampipes/vocabulary/StreamPipes.java | 15 +
.../routing/StandaloneSpInputCollector.java | 2 +-
ui/Dockerfile | 17 --
.../app/configuration/configuration.component.html | 4 +
ui/src/app/configuration/configuration.module.ts | 5 +-
.../consul-configs-boolean.component.ts | 4 +-
.../consul-configs-number.component.ts | 4 +-
.../consul-configs-password.component.ts | 4 +-
.../consul-configs-text.component.ts | 4 +-
.../consul-service.component.spec.ts | 3 +-
.../edge-configuration.component.css | 114 ++++++++
.../edge-configuration.component.html | 67 +++++
.../edge-configuration.component.ts | 67 +++++
.../messaging-configuration.component.html | 20 ++
.../messaging-configuration.component.ts | 4 +
ui/src/app/configuration/model/NodeInfo.model.ts | 97 +++++++
.../pipeline-element-configuration.component.html | 8 +
.../pipeline-element-configuration.component.ts | 6 +-
.../configuration/shared/configuration.service.ts | 13 +-
.../shared/configuration.test.service.ts | 6 +-
.../shared/messaging-settings.model.ts | 1 +
.../shared/streampipes-pe-container-configs.ts | 2 +-
.../shared/streampipes-pe-container.model.ts | 5 +-
.../save-pipeline/save-pipeline.controller.ts | 39 +++
.../save-pipeline/submitPipelineModal.tmpl.html | 43 ++-
ui/src/app/services/rest-api.service.ts | 6 +
129 files changed, 4221 insertions(+), 704 deletions(-)
copy streampipes-config/src/main/java/org/apache/streampipes/config/backend/{SpDataFormat.java => SpProtocol.java} (68%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/AdapterMasterManagementTest.java (100%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/DescriptionManagementTest.java (100%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/management/AdapterTemplateMasterManagementTest.java (100%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/management/SourcesManagementTest.java (100%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/management/UnitMasterManagementTest.java (100%)
rename streampipes-connect-container-master/src/test/java/org/{ => apache}/streampipes/connect/container/master/management/WorkerRestClientTest.java (100%)
create mode 100644 streampipes-connect/src/main/java/org/apache/streampipes/connect/adapter/preprocessing/elements/SendToBrokerAdapterSink.java
copy streampipes-connect/src/main/java/org/apache/streampipes/connect/adapter/preprocessing/elements/{AddTimestampPipelineElement.java => SendToJmsAdapterSink.java} (65%)
copy streampipes-connect/src/main/java/org/apache/streampipes/connect/adapter/preprocessing/elements/{AddTimestampPipelineElement.java => SendToMqttAdapterSink.java} (66%)
create mode 100644 streampipes-container/src/main/java/org/apache/streampipes/container/model/consul/ConsulServiceRegistrationBody.java
copy streampipes-config/src/main/java/org/apache/streampipes/config/model/PeConfig.java => streampipes-container/src/main/java/org/apache/streampipes/container/model/consul/HealthCheckConfiguration.java (53%)
create mode 100644 streampipes-container/src/main/java/org/apache/streampipes/container/util/NodeUtil.java
copy streampipes-backend/src/main/java/org/apache/streampipes/backend/WelcomePageController.java => streampipes-messaging-jms/src/main/java/org/apache/streampipes/messaging/jms/ActiveMQUtils.java (71%)
copy ui/src/app/sensors/components/general/collapsible.controller.ts => streampipes-model/src/main/java/org/apache/streampipes/model/node/Node.java (77%)
copy streampipes-config/src/main/java/org/apache/streampipes/config/model/PeConfig.java => streampipes-model/src/main/java/org/apache/streampipes/model/node/NodeBrokerInfo.java (51%)
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/NodeInfo.java
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/NodeInfoBuilder.java
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/NodeMetadata.java
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/NodeResources.java
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/PipelineElementDockerContainer.java
copy ui/src/app/sensors/components/general/advanced-settings.controller.ts => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/hardware/CPU.java (69%)
copy ui/src/app/editor/components/group/group.controller.ts => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/hardware/DISK.java (76%)
copy streampipes-model/src/main/java/org/apache/streampipes/model/{connect/unit/UnitDescription.java => node/resources/hardware/GPU.java} (60%)
copy streampipes-model/src/main/java/org/apache/streampipes/model/{connect/unit/UnitDescription.java => node/resources/hardware/HardwareResource.java} (57%)
copy ui/src/app/sensors/components/general/collapsible.controller.ts => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/hardware/MEM.java (76%)
copy streampipes-config/src/main/java/org/apache/streampipes/config/model/PeConfig.java => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/interfaces/AccessibleSensorActuatorResource.java (58%)
copy streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/monitoring/task/TaskDefinition.java => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/software/Cuda.java (54%)
create mode 100644 streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/software/Docker.java
copy streampipes-connect/src/main/java/org/apache/streampipes/connect/adapter/guess/PropertyGuesses.java => streampipes-model/src/main/java/org/apache/streampipes/model/node/resources/software/SoftwareResource.java (53%)
copy {streampipes-connect-container-master => streampipes-node-controller-container}/Dockerfile (67%)
copy {streampipes-connect-container-master => streampipes-node-controller-container}/aarch64.Dockerfile (74%)
copy {streampipes-connect-container-master => streampipes-node-controller-container}/arm.Dockerfile (73%)
create mode 100644 streampipes-node-controller-container/development/env
copy {streampipes-backend => streampipes-node-controller-container}/pom.xml (68%)
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/api/NodeControllerResource.java
copy streampipes-connect-container/src/main/java/org/apache/streampipes/connect/rest/AbstractContainerResource.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/api/NodeInfoStatusResource.java (56%)
copy streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/Visualization.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/api/NodeResource.java (71%)
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/config/ConfigKeys.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/config/NodeControllerConfig.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/init/NodeControllerContainer.java
copy streampipes-connect-container/src/main/java/org/apache/streampipes/connect/init/AdapterContainerConfig.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/init/NodeControllerContainerResourceConfig.java (57%)
copy streampipes-commons/src/main/java/org/apache/streampipes/commons/messaging/IMessagePublisher.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/container/ContainerOrchestrator.java (72%)
copy streampipes-commons/src/main/java/org/apache/streampipes/commons/exceptions/NoMatchingProtocolException.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/container/ContainerStatus.java (82%)
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/container/DockerInfo.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/container/DockerOrchestratorManager.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/container/DockerUtils.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/info/NodeInfoStorage.java
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/node/NodeJanitorManager.java
copy streampipes-commons/src/main/java/org/apache/streampipes/commons/exceptions/ElementNotFoundException.java => streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/pe/PipelineElementManager.java (69%)
create mode 100644 streampipes-node-controller-container/src/main/java/org/apache/streampipes/node/controller/container/management/resource/ResourceManager.java
copy {streampipes-connect-container-master => streampipes-node-controller-container}/src/main/resources/banner.txt (90%)
copy streampipes-model-client/src/main/java/org/apache/streampipes/model/client/util/DatasetUtil.java => streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/execution/http/DataSetEntityUrlGenerator.java (61%)
copy streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/{data/PipelineGraph.java => execution/http/EndpointUrlGenerator.java} (63%)
create mode 100644 streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/execution/http/InvocableEntityUrlGenerator.java
create mode 100644 streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/node/AvailableNodesFetcher.java
copy streampipes-rest/src/main/java/org/apache/streampipes/rest/api/{ICouchdb.java => INode.java} (92%)
copy streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/{DataStream.java => Node.java} (70%)
create mode 100644 ui/src/app/configuration/edge-configuration/edge-configuration.component.css
create mode 100644 ui/src/app/configuration/edge-configuration/edge-configuration.component.html
create mode 100644 ui/src/app/configuration/edge-configuration/edge-configuration.component.ts
create mode 100644 ui/src/app/configuration/model/NodeInfo.model.ts