You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by bo...@apache.org on 2022/11/12 15:01:15 UTC
[incubator-streampipes] branch feature/STREAMPIPES-607 created (now f4f1b0410)
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a change to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
at f4f1b0410 tmp
This branch includes the following new commits:
new e8a771bf1 [STREAMPIPES-606] create module for python client
new 5ccd988d6 tmp
new 5d2406553 tmp
new 554f99633 test
new f1c993bbe [STREAMPIPES-622] Modify README intro text, add animated screenshots
new 65ee21591 [STREAMPIPES-622] Improve README structure
new f29670b87 [#STREAMPIPES-622] Modify feature texts
new a418be586 [STREAMPIPES-622] Modify introduction text
new 52ba7ee10 [STREAMPIPES-622] Fix some links in README file
new cd0a54fb2 [hotfix] Update contribution section in README
new fb705eb4a test
new 49830cae5 tmp
new 2c1fe91f1 tmp
new 37c67d015 tmp
new e6151d277 tmp
new 79354a97f tmp
new ce6cde834 tmp
new f4f1b0410 tmp
The 18 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-streampipes] 14/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 37c67d01574e84b6feb5ffe330ab1e5237f11d01
Author: bossenti <bo...@posteo.de>
AuthorDate: Fri Nov 11 21:40:01 2022 +0100
tmp
---
streampipes-client-python/.pre-commit-config.yaml | 31 +++++-----
.../{README.md => DEVELOPMENT.md} | 48 +++++----------
streampipes-client-python/Makefile | 2 +-
streampipes-client-python/README.md | 24 +-------
streampipes-client-python/setup.py | 10 +---
.../streampipes_client/__init__.py | 6 +-
.../streampipes_client/__version__.py | 4 ++
.../streampipes_client/client/__init__.py | 6 +-
.../streampipes_client/client/client.py | 40 ++++++-------
.../streampipes_client/client/credentials.py | 22 +++----
.../client/{ => endpoint}/__init__.py | 9 ++-
.../{ => client}/endpoint/data_lake_measure.py | 3 +-
.../streampipes_client/client/endpoint/endpoint.py | 68 ++++++++++++++++------
.../streampipes_client/endpoint/__init__.py | 16 -----
.../streampipes_client/model/__init__.py | 2 +-
.../streampipes_client/model/common.py | 24 ++++----
.../model/container/data_lake_measures.py | 4 +-
.../model/container/model_container.py | 10 +---
.../streampipes_client/model/element/__init__.py | 2 +-
.../model/element/data_lake_measure.py | 3 +-
20 files changed, 159 insertions(+), 175 deletions(-)
diff --git a/streampipes-client-python/.pre-commit-config.yaml b/streampipes-client-python/.pre-commit-config.yaml
index c876c23f3..334a07141 100644
--- a/streampipes-client-python/.pre-commit-config.yaml
+++ b/streampipes-client-python/.pre-commit-config.yaml
@@ -14,8 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
-files: streampipes_client, tests
+files: streampipes_client/|tests/
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.2.0
@@ -28,32 +27,38 @@ repos:
- repo: local
hooks:
+ - id: pyupgrade
+ name: pyupgrade
+ language: python
+ types: [ python ]
+ entry: pyupgrade --py38-plus --keep-runtime-typing
+ require_serial: true
+
- id: black
name: black
language: python
types: [python]
- entry: black
+ entry: black --line-length=120
+ require_serial: true
- id: isort
name: isort
language: python
types: [python]
- entry: isort
+ entry: isort --profile black
+ require_serial: true
+ verbose: true
- - id: ruff
- name: ruff
+ - id: flake8
+ name: flake8
language: python
types: [python]
- entry: ruff
+ entry: flake8 --max-line-length 120
+ require_serial: true
- id: mypy
name: mypy
language: python
types: [python]
entry: mypy
-
- - id: pyupgrade
- name: pyupgrade
- language: python
- types: [python]
- entry: pyupgrade --py38-plus --keep-runtime-typing
\ No newline at end of file
+ require_serial: true
\ No newline at end of file
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/DEVELOPMENT.md
similarity index 52%
copy from streampipes-client-python/README.md
copy to streampipes-client-python/DEVELOPMENT.md
index c4164d196..27bb7d1f3 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/DEVELOPMENT.md
@@ -16,38 +16,13 @@
~
-->
-# StreamPipes client for Python
+# Development Guide
+This document describes how to easily set up your local dev environment to develop on the
+StreamPipes Python client :snake:
-Apache StreamPipes meets Python! We are working highly motivated on a Python-based client to interact with StreamPipes.
-In this way, we would like to unite the power of StreamPipes to easily connect to and read different data sources, especially in the IoT domain,
-and the amazing universe of data analytics libraries in Python.
+### First Steps :rocket:
-<h1 align="center">
- <br>
- <img src="./docs/img/streampipes-python.png"
- alt="StreamPipes Logo with Python" title="Apache StreamPipes Logo with Python" width="75%"/>
- <br>
-</h1>
-
-<br>
-
-**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
-<br>
-<br>
-**The current version of this Python client is still in alpha phase at best**
-<br>
-**This means that it is still heavily under development, which may result in frequent and extensive API changes, unstable behavior, etc.**
-<br>
-**Please consider it only as paa sneak preview.**
-<br>
-<br>
-**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
-
-<br>
-
-### Get ready for development
-
-1) Set up your Python environment
+1) **Set up your Python environment**
<br>
Create a virtual Python environment with a tool of your choice, here are some examples.
As a next step, install all required dependencies for the development, e.g. with pip:
@@ -56,7 +31,8 @@ As a next step, install all required dependencies for the development, e.g. with
pip install .[dev] # or alternatively: pip install .[docs]
```
<br>
-2) Install pre-commit hook
+
+2) **Install pre-commit hook**
<br>
The pre-commit hook is run before every commit and takes care about code style,
linting, type hints, import sorting, etc.
@@ -65,4 +41,12 @@ Ensure to have the recent version of the pre-commit installed otherwise the CI b
```
pre-commit install
```
-The definition of the pre-commit hook can found in [.pre-commit-config.yaml](.pre-commit-config.yaml).
\ No newline at end of file
+The definition of the pre-commit hook can be found in [.pre-commit-config.yaml](.pre-commit-config.yaml).
+
+### Conventions :clap:
+Below we list some conventions that we have agreed on for creating the StreamPipes Python client.
+Please follow them when you plan to contribute to this project.
+If you have any other suggestions or would like to discuss them, we would be happy to hear from you on our mailing list [dev@streampipes.apache.org](mailto:dev@streampipes.apache.org).
+
+1) **Use `numpy` style for Python docstrings** :page_facing_up: <br>
+Please stick to the `numpy` style when writing docstrings, as we require this for generating our documentation.
diff --git a/streampipes-client-python/Makefile b/streampipes-client-python/Makefile
index 309ad627f..55a203d71 100644
--- a/streampipes-client-python/Makefile
+++ b/streampipes-client-python/Makefile
@@ -27,4 +27,4 @@ livedoc: doc
mkdocs serve --dirtyreload
rebase:
- git fetch && git rebase origin/dev
\ No newline at end of file
+ git fetch && git rebase origin/dev
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index c4164d196..8cad0839d 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -24,7 +24,7 @@ and the amazing universe of data analytics libraries in Python.
<h1 align="center">
<br>
- <img src="./docs/img/streampipes-python.png"
+ <img src="./docs/img/streampipes-python.png"
alt="StreamPipes Logo with Python" title="Apache StreamPipes Logo with Python" width="75%"/>
<br>
</h1>
@@ -44,25 +44,3 @@ and the amazing universe of data analytics libraries in Python.
**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
<br>
-
-### Get ready for development
-
-1) Set up your Python environment
-<br>
-Create a virtual Python environment with a tool of your choice, here are some examples.
-As a next step, install all required dependencies for the development, e.g. with pip:
-
-```
-pip install .[dev] # or alternatively: pip install .[docs]
-```
-<br>
-2) Install pre-commit hook
-<br>
-The pre-commit hook is run before every commit and takes care about code style,
-linting, type hints, import sorting, etc.
-Ensure to have the recent version of the pre-commit installed otherwise the CI build might fail:
-
-```
-pre-commit install
-```
-The definition of the pre-commit hook can found in [.pre-commit-config.yaml](.pre-commit-config.yaml).
\ No newline at end of file
diff --git a/streampipes-client-python/setup.py b/streampipes-client-python/setup.py
index 4914564f8..9cc3766d1 100644
--- a/streampipes-client-python/setup.py
+++ b/streampipes-client-python/setup.py
@@ -30,23 +30,19 @@ AUTHOR = "Apache Software Foundation"
REQUIRES_PYTHON = ">=3.8.0"
# Package requirements.
-base_packages = [
- "pandas>=1.5.1",
- "pydantic>=1.10.2",
- "requests>=2.28.1"
-]
+base_packages = ["pandas>=1.5.1", "pydantic>=1.10.2", "requests>=2.28.1"]
dev_packages = base_packages + [
"mypy>=0.990",
- "pre-commit>=2.20.0",
"pytest>=7.2.0",
"pytest-cov>=4.0.0",
"black>=22.10.0",
"flake8>=5.0.4",
"isort>=5.10.1",
+ "pyupgrade>=3.2.2"
]
-docs_packages = [ #todo Cedric check versions
+docs_packages = [
"mkdocs>=1.2.3",
"mkdocs-material>=8.1.11",
"mkdocstrings[python]>=0.19.0",
diff --git a/streampipes-client-python/streampipes_client/__init__.py b/streampipes-client-python/streampipes_client/__init__.py
index ecb1860df..7bb805d49 100644
--- a/streampipes-client-python/streampipes_client/__init__.py
+++ b/streampipes-client-python/streampipes_client/__init__.py
@@ -13,4 +13,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
\ No newline at end of file
+#
+
+"""
+This library provides a handy Python-based client to interact with Apache StreamPipes.
+"""
diff --git a/streampipes-client-python/streampipes_client/__version__.py b/streampipes-client-python/streampipes_client/__version__.py
index 492084802..819a48a81 100644
--- a/streampipes-client-python/streampipes_client/__version__.py
+++ b/streampipes-client-python/streampipes_client/__version__.py
@@ -15,6 +15,10 @@
# limitations under the License.
#
+"""
+Storing the current version number, predominantly used by `setup.py`.
+"""
+
VERSION = (0, 71, 0)
__version__ = ".".join(map(str, VERSION)) # noqa: F401
diff --git a/streampipes-client-python/streampipes_client/client/__init__.py b/streampipes-client-python/streampipes_client/client/__init__.py
index a323247a5..47db35729 100644
--- a/streampipes-client-python/streampipes_client/client/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/__init__.py
@@ -15,5 +15,9 @@
# limitations under the License.
#
-from .client import StreamPipesClient
+"""
+This module contains everything about the client itself, the central point of interaction with StreamPipes.
+"""
+
+from .client import StreamPipesClient, StreamPipesClientConfig
from .credentials import StreamPipesApiKeyCredentials
diff --git a/streampipes-client-python/streampipes_client/client/client.py b/streampipes-client-python/streampipes_client/client/client.py
index f7948b7c0..51323d947 100644
--- a/streampipes-client-python/streampipes_client/client/client.py
+++ b/streampipes-client-python/streampipes_client/client/client.py
@@ -20,18 +20,15 @@ from dataclasses import dataclass
from typing import Dict, Optional
import requests
-
from streampipes_client.client.credentials import CredentialProvider
-from streampipesclient.endpoints import DataLakeMeasureEndpoint
+from streampipes_client.client.endpoint import DataLakeMeasureEndpoint
logger = logging.getLogger(__name__)
@dataclass
class StreamPipesClientConfig:
- """
-
- """
+ """ """
credential_provider: CredentialProvider
host_address: str
@@ -40,12 +37,12 @@ class StreamPipesClientConfig:
class StreamPipesClient:
-
- def __init__(self,
- *,
- client_config: StreamPipesClientConfig,
- logging_level: int = logging.INFO,
- ):
+ def __init__(
+ self,
+ *,
+ client_config: StreamPipesClientConfig,
+ logging_level: int = logging.INFO,
+ ):
self.client_config = client_config
self.request_session = requests.Session()
@@ -67,19 +64,22 @@ class StreamPipesClient:
logger.info(f"Logging successfully initialized with logging level {logging.getLevelName(logging_level)}.")
@classmethod
- def create(cls, *, client_config: StreamPipesClientConfig, logging_level: int = logging.INFO):
+ def create(
+ cls,
+ *,
+ client_config: StreamPipesClientConfig,
+ logging_level: int = logging.INFO,
+ ):
return cls(client_config=client_config, logging_level=logging_level)
@property
def http_headers(self) -> Dict[str, str]:
- return self.client_config.credential_provider.make_headers(
- {
- "Application": "application/json"
- }
- )
+ return self.client_config.credential_provider.make_headers({"Application": "application/json"})
@property
def base_api_path(self) -> str:
- return f"{'http://' if self.client_config.https_disabled else 'https://'}" \
- f"{self.client_config.host_address}:" \
- f"{self.client_config.port}/streampipes-backend/"
+ return (
+ f"{'http://' if self.client_config.https_disabled else 'https://'}"
+ f"{self.client_config.host_address}:"
+ f"{self.client_config.port}/streampipes-backend/"
+ )
diff --git a/streampipes-client-python/streampipes_client/client/credentials.py b/streampipes-client-python/streampipes_client/client/credentials.py
index fe2cc2b89..de60cb95b 100644
--- a/streampipes-client-python/streampipes_client/client/credentials.py
+++ b/streampipes-client-python/streampipes_client/client/credentials.py
@@ -19,32 +19,26 @@ from typing import Dict, Optional
class CredentialProvider(ABC):
-
@abstractmethod
def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
raise NotImplementedError
class StreamPipesApiKeyCredentials(CredentialProvider):
-
- def __init__(self,
- *,
- username: str,
- api_key: str,
- ):
+ def __init__(
+ self,
+ *,
+ username: str,
+ api_key: str,
+ ):
self.username = username
self.api_key = api_key
- def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
+ def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
if http_headers is None:
http_headers = {}
- http_headers.update(
- {
- "X-API-User": self.username,
- "X-API-Key": self.api_key
- }
- )
+ http_headers.update({"X-API-User": self.username, "X-API-Key": self.api_key})
return http_headers
diff --git a/streampipes-client-python/streampipes_client/client/__init__.py b/streampipes-client-python/streampipes_client/client/endpoint/__init__.py
similarity index 70%
copy from streampipes-client-python/streampipes_client/client/__init__.py
copy to streampipes-client-python/streampipes_client/client/endpoint/__init__.py
index a323247a5..75f4dca5d 100644
--- a/streampipes-client-python/streampipes_client/client/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/__init__.py
@@ -14,6 +14,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+"""
+This module contains the implementation of all endpoints of the StreamPipes client.
+One endpoint aims to handle the whole interaction with a central endpoint of the StreamPipes API.
+"""
+from .data_lake_measure import DataLakeMeasureEndpoint
+from .endpoint import APIEndpoint
-from .client import StreamPipesClient
-from .credentials import StreamPipesApiKeyCredentials
+__all__ = ["APIEndpoint", "DataLakeMeasureEndpoint"]
diff --git a/streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
similarity index 83%
rename from streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py
rename to streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
index 6ee4c8403..76f5a1c2a 100644
--- a/streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
@@ -1,9 +1,8 @@
-from streampipes_client.endpoint.endpoint import APIEndpoint
+from client.endpoint.endpoint import APIEndpoint
from streampipes_client.model.container.data_lake_measures import DataLakeMeasures
class DataLakeMeasureEndpoint(APIEndpoint):
-
@classmethod
def _container_cls(cls):
return DataLakeMeasures
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index 9bf2f1984..8547c69bf 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -21,37 +21,52 @@ Provided classes and assets are aimed to be used for developing endpoints.
An endpoint is provides all options to communicate with a central endpoint of the StreamPipes API in a handy way.
"""
-from http import HTTPStatus
import logging
from abc import ABC, abstractmethod
-from typing import Tuple, Type, Callable
+from http import HTTPStatus
+from typing import Callable, Tuple, Type
from requests import Response
-from requests.exceptions import HTTPError
-
+from requests.exceptions import BaseHTTPError, HTTPError
from streampipes_client.model.container import ModelContainer
from streampipes_client.model.element import Element
+__all__ = ["APIEndpoint"]
+
logger = logging.getLogger(__name__)
# define custom logging messages for some specific HTTP status
status_code_to_log_message = {
401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
- "Please check your user name and/or password to be correct.",
- 403: "\nThere seems to be an issue with the access rights of the given user and the ressource you queried.\n"
- "Apparently, this user is not allowed to query the resource.\n"
- "Please check the user's permissions or contact your StreamPipes admin.",
+ "Please check your user name and/or password to be correct.",
+ 403: "\nThere seems to be an issue with the access rights of the given user and the resource you queried.\n"
+ "Apparently, this user is not allowed to query the resource.\n"
+ "Please check the user's permissions or contact your StreamPipes admin.",
**dict.fromkeys(
[404, 405],
"\nOops, there seems to be an issue with the Python Client calling the API inappropriately.\n"
"This should not happen, but unfortunately did.\n"
- "If you don't mind, it would be awesome to let us know by creating an issue at github.com/apache/incubator-streampipes.\n"
- "Please paste the following information to the issue description:\n\n")
+ "If you don't mind, it would be awesome to let us know by creating an issue"
+ " at github.com/apache/incubator-streampipes.\n"
+ "Please paste the following information to the issue description:\n\n",
+ ),
}
class APIEndpoint(ABC):
+ """Abstract implementation of an API endpoint.
+ Serves as template for all endpoints for the StreamPipes API.
+ By design, endpoints are only instantiated within the `__init__` method of the StreamPipesClient.
+
+ Could be extended by additional subtypes in the future.
+
+ Parameters
+ ----------
+ parent_client: `client.StreamPipesClient`
+ This parameter expects the instance of the `StreamPipesClient` the endpoint is attached to.
+
+ """
def __init__(self, parent_client: "StreamPipesClient"):
self._parent_client = parent_client
@@ -59,18 +74,34 @@ class APIEndpoint(ABC):
@property
@abstractmethod
def _relative_api_path(self) -> Tuple[str]:
+ """
+ Defines the relative api path with regard to the StreamPipes API URL.
+ Each path within the URL is defined as an own string.
+
+ Returns
+ -------
+ A tuple of strings of which every represents a path value of the endpoint's API URL.
+
+ """
raise NotImplementedError
@classmethod
@abstractmethod
def _container_cls(cls) -> Type[ModelContainer]:
+ """
+
+ Returns
+ -------
+ A class which inherits from `model.ModelContainer`
+ """
raise NotImplementedError
- def _make_request(self,
- *,
- request_method: Callable[..., Response],
- url: str,
- ) -> Response:
+ def _make_request(
+ self,
+ *,
+ request_method: Callable[..., Response],
+ url: str,
+ ) -> Response:
response = request_method(url=url)
@@ -82,7 +113,10 @@ class APIEndpoint(ABC):
log_message = status_code_to_log_message[err.response.status_code]
- if status_code in [HTTPStatus.METHOD_NOT_ALLOWED.numerator, HTTPStatus.NOT_FOUND.numerator]:
+ if status_code in [
+ HTTPStatus.METHOD_NOT_ALLOWED.numerator,
+ HTTPStatus.NOT_FOUND.numerator,
+ ]:
log_message += f"url: {err.response.url}\nstatus code: {err.response.status_code}"
logger.debug(err.response.text)
@@ -101,7 +135,7 @@ class APIEndpoint(ABC):
response = self._make_request(
request_method=self._parent_client.request_session.get,
- url=self.create_api_path()
+ url=self.create_api_path(),
)
return self._container_cls().from_json(json_string=response.text)
diff --git a/streampipes-client-python/streampipes_client/endpoint/__init__.py b/streampipes-client-python/streampipes_client/endpoint/__init__.py
deleted file mode 100644
index ecb1860df..000000000
--- a/streampipes-client-python/streampipes_client/endpoint/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
\ No newline at end of file
diff --git a/streampipes-client-python/streampipes_client/model/__init__.py b/streampipes-client-python/streampipes_client/model/__init__.py
index ecb1860df..cce3acad3 100644
--- a/streampipes-client-python/streampipes_client/model/__init__.py
+++ b/streampipes-client-python/streampipes_client/model/__init__.py
@@ -13,4 +13,4 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
\ No newline at end of file
+#
diff --git a/streampipes-client-python/streampipes_client/model/common.py b/streampipes-client-python/streampipes_client/model/common.py
index b1994372f..4c78c906c 100644
--- a/streampipes-client-python/streampipes_client/model/common.py
+++ b/streampipes-client-python/streampipes_client/model/common.py
@@ -17,7 +17,7 @@
from typing import List, Optional
-from pydantic import BaseModel, StrictStr, StrictInt, StrictBool, ValidationError
+from pydantic import BaseModel, StrictBool, StrictInt, StrictStr, ValidationError
def snake_to_camel_case(snake_case_string: str) -> str:
@@ -67,19 +67,21 @@ class EventSchema(BasicModel):
class StreamPipesDataModelError(RuntimeError):
-
def __init__(self, validation_error: ValidationError):
super().__init__(
- self._generate_error_message(model=validation_error.model,
- error_description=validation_error.json(),
- )
+ self._generate_error_message(
+ model=validation_error.model,
+ error_description=validation_error.json(),
+ )
)
@staticmethod
def _generate_error_message(*, model: BasicModel, error_description: str) -> str:
- return f"\nOops, there seems to be a problem with our internal StreamPipes data model.\n" \
- f"This should not occur, but unfortunately did.\n" \
- f"Therefore, it would be great if you could report this problem as an issue at github.com/apache/incubator-streampipes.\n" \
- f"Please don't forget to include the following information:\n\n" \
- f"Affected Model class: {str(model)}\n" \
- f"Validation error log: {error_description}"
+ return (
+ f"\nOops, there seems to be a problem with our internal StreamPipes data model.\n"
+ f"This should not occur, but unfortunately did.\n"
+ f"Therefore, it would be great if you could report this problem as an issue at github.com/apache/incubator-streampipes.\n"
+ f"Please don't forget to include the following information:\n\n"
+ f"Affected Model class: {str(model)}\n"
+ f"Validation error log: {error_description}"
+ )
diff --git a/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
index 758a57a8b..9f36cf86b 100644
--- a/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
+++ b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
@@ -1,14 +1,12 @@
import pandas as pd
-
from streampipes_client.model.container import ModelContainer
from streampipes_client.model.element.data_lake_measure import DataLakeMeasure
class DataLakeMeasures(ModelContainer):
-
@classmethod
def _element_cls(cls):
return DataLakeMeasure
def to_pandas(self) -> pd.DataFrame:
- pass
\ No newline at end of file
+ pass
diff --git a/streampipes-client-python/streampipes_client/model/container/model_container.py b/streampipes-client-python/streampipes_client/model/container/model_container.py
index 072e590ff..596545404 100644
--- a/streampipes-client-python/streampipes_client/model/container/model_container.py
+++ b/streampipes-client-python/streampipes_client/model/container/model_container.py
@@ -15,20 +15,16 @@
# limitations under the License.
#
import json
-
-import pandas as pd
-
from abc import ABC, abstractmethod
from typing import List, Type
+import pandas as pd
from pydantic import ValidationError
-
from streampipes_client.model.common import StreamPipesDataModelError
from streampipes_client.model.element import Element
class ModelContainer(ABC):
-
def __init__(self, elements: List[Element]):
self._elements = elements
@@ -48,9 +44,7 @@ class ModelContainer(ABC):
try:
- model_container = cls(
- elements=[cls._element_cls().parse_obj(list_item) for list_item in data]
- )
+ model_container = cls(elements=[cls._element_cls().parse_obj(list_item) for list_item in data])
except ValidationError as ve:
raise StreamPipesDataModelError(validation_error=ve)
diff --git a/streampipes-client-python/streampipes_client/model/element/__init__.py b/streampipes-client-python/streampipes_client/model/element/__init__.py
index f4b8a3ece..a8f3b03aa 100644
--- a/streampipes-client-python/streampipes_client/model/element/__init__.py
+++ b/streampipes-client-python/streampipes_client/model/element/__init__.py
@@ -15,4 +15,4 @@
# limitations under the License.
#
-from .element import Element
\ No newline at end of file
+from .element import Element
diff --git a/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py b/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
index 444978d3e..ca5a89fd9 100644
--- a/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
@@ -16,8 +16,7 @@
#
from typing import Optional
-from pydantic import StrictStr, StrictBool
-
+from pydantic import StrictBool, StrictStr
from streampipes_client.model.common import EventSchema
from streampipes_client.model.element import Element
[incubator-streampipes] 18/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit f4f1b0410645d827072e044dc76f5b7bab998bbb
Author: bossenti <bo...@posteo.de>
AuthorDate: Sat Nov 12 16:00:55 2022 +0100
tmp
---
.../streampipes_client/__init__.py | 10 +-
.../streampipes_client/client/__init__.py | 2 +-
.../streampipes_client/client/client.py | 10 +-
.../client/credential_provider.py | 111 +++++++++++++++
.../client/endpoint/data_lake_measure.py | 9 +-
.../streampipes_client/client/endpoint/endpoint.py | 20 +--
.../streampipes_client/model/common.py | 58 ++++----
.../streampipes_client/model/container/__init__.py | 13 +-
.../model/container/data_lake_measures.py | 59 +++++++-
.../model/container/model_container.py | 60 --------
.../model/container/resource_container.py | 151 +++++++++++++++++++++
.../streampipes_client/model/element/__init__.py | 18 ---
.../streampipes_client/model/exception.py | 86 ++++++++++++
.../{element/element.py => resource/__init__.py} | 11 +-
.../{element => resource}/data_lake_measure.py | 18 ++-
.../credentials.py => model/resource/resource.py} | 40 +++---
16 files changed, 502 insertions(+), 174 deletions(-)
diff --git a/streampipes-client-python/streampipes_client/__init__.py b/streampipes-client-python/streampipes_client/__init__.py
index 285906ad1..95500c32f 100644
--- a/streampipes-client-python/streampipes_client/__init__.py
+++ b/streampipes-client-python/streampipes_client/__init__.py
@@ -17,12 +17,4 @@
"""
This library provides a handy Python-based client to interact with Apache StreamPipes.
-"""
-
-from .client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
-
-__all = [
- "StreamPipesApiKeyCredentials"
- "StreamPipesClient",
- "StreamPipesClientConfig",
-]
+"""
\ No newline at end of file
diff --git a/streampipes-client-python/streampipes_client/client/__init__.py b/streampipes-client-python/streampipes_client/client/__init__.py
index 98d1d0e37..35f6c8530 100644
--- a/streampipes-client-python/streampipes_client/client/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/__init__.py
@@ -20,7 +20,7 @@ This module contains everything about the client itself, the central point of in
"""
from .client import StreamPipesClient, StreamPipesClientConfig
-from .credentials import StreamPipesApiKeyCredentials
+from .credential_provider import StreamPipesApiKeyCredentials
__all__ = [
"StreamPipesApiKeyCredentials",
diff --git a/streampipes-client-python/streampipes_client/client/client.py b/streampipes-client-python/streampipes_client/client/client.py
index 9e7e2b471..91ae647bd 100644
--- a/streampipes-client-python/streampipes_client/client/client.py
+++ b/streampipes-client-python/streampipes_client/client/client.py
@@ -21,13 +21,15 @@ The client is designed as the central point of interaction with the StreamPipes
provides all functionalities to communicate with the API.
"""
+from __future__ import annotations
+
import logging
import sys
from dataclasses import dataclass
from typing import Dict, Optional
import requests
-from streampipes_client.client.credentials import CredentialProvider
+from streampipes_client.client.credential_provider import CredentialProvider
from streampipes_client.client.endpoint import DataLakeMeasureEndpoint
logger = logging.getLogger(__name__)
@@ -80,7 +82,7 @@ class StreamPipesClient:
Examples
--------
- >>> from streampipes_client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
+ >>> from streampipes_client.client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
>>> client_config = StreamPipesClientConfig(
... credential_provider=StreamPipesApiKeyCredentials(
@@ -144,7 +146,7 @@ class StreamPipesClient:
cls,
client_config: StreamPipesClientConfig,
logging_level: int = logging.INFO,
- ):
+ ) -> StreamPipesClient:
"""Returns an instance of the `StreamPipesPythonClient`.
Provides consistency to the Java client.
@@ -169,7 +171,7 @@ class StreamPipesClient:
Returns
-------
- Dictionary with header information as string key-value pairs
+ Dictionary with header information as string key-value pairs.
"""
# create HTTP headers from credential provider and add additional headers needed
diff --git a/streampipes-client-python/streampipes_client/client/credential_provider.py b/streampipes-client-python/streampipes_client/client/credential_provider.py
new file mode 100644
index 000000000..dd99dd6ba
--- /dev/null
+++ b/streampipes-client-python/streampipes_client/client/credential_provider.py
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of credential providers.
+A credential provider supplies the specified sort of credentials in the appropriate HTTP header format.
+The headers are then used by the client to connect to StreamPipes.
+"""
+
+from abc import ABC, abstractmethod
+from typing import Dict, Optional
+
+__all__ = [
+ "StreamPipesApiKeyCredentials",
+]
+
+
+class CredentialProvider(ABC):
+ """Abstract implementation of a credential provider.
+ Must be inherited by all credential providers.
+ """
+
+ def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
+ """Creates the HTTP headers for the specific credential provider.
+ Concrete authentication headers must be defined in the implementation of a credential provider.
+
+ Parameters
+ ----------
+ http_headers: Optional[Dict[str, str]]
+ Additional HTTP headers the generated headers are extended by.
+
+ Returns
+ -------
+ Dictionary with header information as string key-value pairs.
+
+ """
+ if http_headers is None:
+ http_headers = {}
+
+ http_headers.update(self._authentication_headers)
+
+ return http_headers
+
+ @property
+ @abstractmethod
+ def _authentication_headers(self) -> Dict[str, str]:
+ """Provides the HTTP headers used for the authentication with the concrete `CredentialProvider`.
+
+ Returns
+ -------
+ Dictionary with authentication headers as string key-value pairs.
+
+ """
+ raise NotImplementedError
+
+
+class StreamPipesApiKeyCredentials(CredentialProvider):
+ """A Credential provider that allows authentication via a StreamPipes API Token.
+ This token can be generated via the StreamPipes UI (see how in the project's README).
+
+ Parameters
+ ----------
+ username: str
+ The username to which the API token is granted, e.g., `demo-user@streampipes.apche.org`.
+ api_key: str
+ The StreamPipes API key as it is displayed in the UI.
+
+ Examples
+ --------
+ see `StreamPipesClient`
+
+ References
+ ----------
+ [^1]: [StreamPipes Python Client README](https://github.com/apache/incubator-streampipes/blob/dev/streampipes-client-python/README.md#%EF%B8%8F-quickstart)
+ """
+
+ def __init__(
+ self,
+ username: str,
+ api_key: str,
+ ):
+ self.username = username
+ self.api_key = api_key
+
+ @property
+ def _authentication_headers(self) -> Dict[str, str]:
+ """Provides the HTTP headers used for the authentication with the API token.
+
+ Returns
+ -------
+ Dictionary with authentication headers as string key-value pairs.
+
+ """
+ return {
+ "X-API-User": self.username,
+ "X-API-Key": self.api_key,
+ }
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
index a46338165..c16d1774a 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
@@ -19,7 +19,10 @@
Specific implementation of the StreamPipes API's data lake measure endpoints.
This endpoint allows to consume data stored in StreamPipes' data lake
"""
+from typing import Tuple, Type
+
from client.endpoint.endpoint import APIEndpoint
+from model.container import ResourceContainer
from streampipes_client.model.container.data_lake_measures import DataLakeMeasures
__all__ = [
@@ -45,7 +48,7 @@ class DataLakeMeasureEndpoint(APIEndpoint):
Examples
--------
- >>> from streampipes_client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
+ >>> from streampipes_client.client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
>>> client_config = StreamPipesClientConfig(
... credential_provider=StreamPipesApiKeyCredentials(username="test-user", api_key="api-key"),
@@ -63,7 +66,7 @@ class DataLakeMeasureEndpoint(APIEndpoint):
"""
@property
- def _container_cls(self):
+ def _container_cls(self) -> Type[ResourceContainer]:
"""Defines the model container class the endpoint refers to.
@@ -74,7 +77,7 @@ class DataLakeMeasureEndpoint(APIEndpoint):
return DataLakeMeasures
@property
- def _relative_api_path(self):
+ def _relative_api_path(self) -> Tuple[str, ...]:
"""Defines the relative api path to the DataLakeMeasurement endpoint.
Each path within the URL is defined as an own string.
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index 7f349b386..f897d6bcf 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -28,8 +28,8 @@ from typing import Callable, Tuple, Type, Union
from requests import Response
from requests.exceptions import BaseHTTPError, HTTPError
-from streampipes_client.model.container import ModelContainer
-from streampipes_client.model.element import Element
+from streampipes_client.model.container import ResourceContainer
+from streampipes_client.model.resource import Resource
__all__ = [
"APIEndpoint",
@@ -74,7 +74,7 @@ class APIEndpoint(ABC):
@property
@abstractmethod
- def _container_cls(cls) -> Type[ModelContainer]:
+ def _container_cls(self) -> Type[ResourceContainer]:
"""Defines the model container class the endpoint refers to.
This model container class corresponds to the Python data model,
which handles multiple resources returned from the endpoint.
@@ -82,13 +82,13 @@ class APIEndpoint(ABC):
Returns
-------
The corresponding container class from the data model,
- needs to a subclass of `model.ModelContainer`.
+ needs to a subclass of `model.container.ResourceContainer`.
"""
raise NotImplementedError
@property
@abstractmethod
- def _relative_api_path(self) -> Tuple[str]:
+ def _relative_api_path(self) -> Tuple[str, ...]:
"""Defines the relative api path with regard to the StreamPipes API URL.
Each path within the URL is defined as an own string.
@@ -164,23 +164,23 @@ class APIEndpoint(ABC):
"""
return f"{self._parent_client.base_api_path}" f"{'/'.join(api_path for api_path in self._relative_api_path)}"
- def all(self) -> ModelContainer:
+ def all(self) -> ResourceContainer:
"""Get all resources of this endpoint provided by the StreamPipes API.
- Results are provided as an instance of a `model.ModelContainer` that
+ Results are provided as an instance of a `model.container.ResourceContainer` that
allows to handle the returned resources in a comfortable and pythonic way.
Returns
-------
- A model container instance (`model.ModelContainer`) bundling the resources returned.
+ A model container instance (`model.container.ResourceContainer`) bundling the resources returned.
"""
response = self._make_request(
request_method=self._parent_client.request_session.get,
url=self.build_url(),
)
- return self._container_cls().from_json(json_string=response.text)
+ return self._container_cls.from_json(json_string=response.text)
- def get(self, identifier: str) -> Element:
+ def get(self, identifier: str) -> Resource:
"""Queries the specified resource from the API endpoint.
Parameters
diff --git a/streampipes-client-python/streampipes_client/model/common.py b/streampipes-client-python/streampipes_client/model/common.py
index 4c78c906c..940d0def7 100644
--- a/streampipes-client-python/streampipes_client/model/common.py
+++ b/streampipes-client-python/streampipes_client/model/common.py
@@ -15,14 +15,22 @@
# limitations under the License.
#
+"""
+Classes of the StreamPipes data model that are commonly shared.
+"""
+
from typing import List, Optional
-from pydantic import BaseModel, StrictBool, StrictInt, StrictStr, ValidationError
+from pydantic import BaseModel, StrictBool, StrictInt, StrictStr, Field
+__all__ = [
+ "BasicModel",
+ "EventSchema"
+]
-def snake_to_camel_case(snake_case_string: str) -> str:
- """
- Converts a string in snake_case format to camelCase style.
+
+def _snake_to_camel_case(snake_case_string: str) -> str:
+ """Converts a string in snake_case format to camelCase style.
"""
tokens = snake_case_string.split("_")
@@ -34,25 +42,27 @@ class BasicModel(BaseModel):
element_id: Optional[StrictStr]
class Config:
- alias_generator = snake_to_camel_case
-
-
-class EventPropertyQualityDefinition(BasicModel):
- pass
+ alias_generator = _snake_to_camel_case
class EventPropertyQualityRequirement(BasicModel):
- minimum_property_quality: Optional[EventPropertyQualityDefinition]
- maximum_property_quality: Optional[EventPropertyQualityDefinition]
+ """
+ Data model of an `EventPropertyQualityRequirement` in compliance to the StreamPipes Backend.
+ """
+ minimum_property_quality: Optional[BasicModel] = Field(alias="EventPropertyQualityDefinition")
+ maximum_property_quality: Optional[BasicModel] = Field(alias="EventPropertyQualityDefinition")
class EventProperty(BasicModel):
+ """
+ Data model of an `EventProperty` in compliance to the StreamPipes Backend.
+ """
label: StrictStr
description: StrictStr
runtime_name: StrictStr
required: StrictBool
domain_properties: List[StrictStr]
- event_property_qualities: List[EventPropertyQualityDefinition]
+ event_property_qualities: List[BasicModel] = Field(alias="EventPropertyQualityDefinition")
requires_event_property_qualities: List[EventPropertyQualityRequirement]
property_scope: Optional[StrictStr]
index: StrictInt
@@ -63,25 +73,7 @@ class EventProperty(BasicModel):
class EventSchema(BasicModel):
+ """
+ Data model of an `EventSchema` in compliance to the StreamPipes Backend.
+ """
event_properties: List[EventProperty]
-
-
-class StreamPipesDataModelError(RuntimeError):
- def __init__(self, validation_error: ValidationError):
- super().__init__(
- self._generate_error_message(
- model=validation_error.model,
- error_description=validation_error.json(),
- )
- )
-
- @staticmethod
- def _generate_error_message(*, model: BasicModel, error_description: str) -> str:
- return (
- f"\nOops, there seems to be a problem with our internal StreamPipes data model.\n"
- f"This should not occur, but unfortunately did.\n"
- f"Therefore, it would be great if you could report this problem as an issue at github.com/apache/incubator-streampipes.\n"
- f"Please don't forget to include the following information:\n\n"
- f"Affected Model class: {str(model)}\n"
- f"Validation error log: {error_description}"
- )
diff --git a/streampipes-client-python/streampipes_client/model/container/__init__.py b/streampipes-client-python/streampipes_client/model/container/__init__.py
index aa5f2a267..ba22996ea 100644
--- a/streampipes-client-python/streampipes_client/model/container/__init__.py
+++ b/streampipes-client-python/streampipes_client/model/container/__init__.py
@@ -15,5 +15,16 @@
# limitations under the License.
#
-from .model_container import ModelContainer
+"""
+This model contains the implementation of all resource container.
+A ResourceContainer is a collection of `model.resource.Resource`s to which the
+response of the StreamPipes API is parsed.
+"""
+
+from .resource_container import ResourceContainer
from .data_lake_measures import DataLakeMeasures
+
+__all__ = [
+ "DataLakeMeasures",
+ "ResourceContainer",
+]
diff --git a/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
index 9f36cf86b..ddd8a5756 100644
--- a/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
+++ b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
@@ -1,12 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Implementation of a resource container for the data lake measures endpoint.
+"""
+
import pandas as pd
-from streampipes_client.model.container import ModelContainer
-from streampipes_client.model.element.data_lake_measure import DataLakeMeasure
+from streampipes_client.model.container import ResourceContainer
+from streampipes_client.model.resource.data_lake_measure import DataLakeMeasure
+
+__all__ = [
+ "DataLakeMeasures",
+]
+
+class DataLakeMeasures(ResourceContainer):
+ """Implementation of the resource container for the data lake measures endpoint.
+ This resource container is a collection of data lake measures returned by the StreamPipes API.
+ It is capable of parsing the response content directly into a list of queried `DataLakeMeasure`.
+ Furthermore, the resource container makes them accessible in a pythonic manner.
+
+ Parameters
+ ----------
+ resources: List[DataLakeMeasure]
+ A list of resources (`model.resource.DataLakeMeasure`) to be contained in the `ResourceContainer`.
+
+ """
-class DataLakeMeasures(ModelContainer):
@classmethod
- def _element_cls(cls):
+ def _resource_cls(cls):
+ """Returns the class of the resource that are bundled.
+
+ Returns
+ -------
+ DataLakeMeasure
+ """
return DataLakeMeasure
def to_pandas(self) -> pd.DataFrame:
- pass
+ """Returns the all data lake measures in one Pandas Dataframe.
+
+ Returns
+ -------
+ pd.DataFrame
+ """
+ raise NotImplementedError
diff --git a/streampipes-client-python/streampipes_client/model/container/model_container.py b/streampipes-client-python/streampipes_client/model/container/model_container.py
deleted file mode 100644
index 596545404..000000000
--- a/streampipes-client-python/streampipes_client/model/container/model_container.py
+++ /dev/null
@@ -1,60 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-import json
-from abc import ABC, abstractmethod
-from typing import List, Type
-
-import pandas as pd
-from pydantic import ValidationError
-from streampipes_client.model.common import StreamPipesDataModelError
-from streampipes_client.model.element import Element
-
-
-class ModelContainer(ABC):
- def __init__(self, elements: List[Element]):
- self._elements = elements
-
- def __getitem__(self, position: int) -> Element:
- return self._elements[position]
-
- def __len__(self) -> int:
- return len(self._elements)
-
- @classmethod
- def from_json(cls, json_string: str) -> "ModelContainer":
-
- data = json.loads(json_string)
-
- if not type(data) == list:
- raise RuntimeError
-
- try:
-
- model_container = cls(elements=[cls._element_cls().parse_obj(list_item) for list_item in data])
- except ValidationError as ve:
- raise StreamPipesDataModelError(validation_error=ve)
-
- return model_container
-
- @abstractmethod
- def to_pandas(self) -> pd.DataFrame:
- raise NotImplementedError
-
- @classmethod
- @abstractmethod
- def _element_cls(cls) -> Type[Element]:
- raise NotImplementedError
diff --git a/streampipes-client-python/streampipes_client/model/container/resource_container.py b/streampipes-client-python/streampipes_client/model/container/resource_container.py
new file mode 100644
index 000000000..20a64b264
--- /dev/null
+++ b/streampipes-client-python/streampipes_client/model/container/resource_container.py
@@ -0,0 +1,151 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+General and abstract implementation for a resource container.
+A resource container is a collection of resources returned by the StreamPipes API.
+It is capable of parsing the response content directly into a list of queried resources.
+Furthermore, the resource container makes them accessible in a pythonic manner.
+"""
+
+from __future__ import annotations
+
+import json
+from abc import ABC, abstractmethod
+from typing import List, Type, Dict
+
+import pandas as pd
+from pydantic import ValidationError
+from model.exception import StreamPipesDataModelError, StreamPipesResourceContainerJSONError
+from streampipes_client.model.resource import Resource
+
+__all__ = [
+ "ResourceContainer"
+]
+
+
+class ResourceContainer(ABC):
+ """General and abstract implementation for a resource container.
+ A resource container is a collection of resources returned by the StreamPipes API.
+ It is capable of parsing the response content directly into a list of queried resources.
+ Furthermore, the resource container makes them accessible in a pythonic manner.
+
+ Parameters
+ ----------
+ resources: List[Resource]
+ A list of resources (`model.resource.Resource`) to be contained in the `ResourceContainer`.
+
+ """
+ def __init__(self, resources: List[Resource]):
+ self._resources = resources
+
+ def __getitem__(self, position: int) -> Resource:
+ return self._resources[position]
+
+ def __len__(self) -> int:
+ return len(self._resources)
+
+ def __repr__(self):
+ new_line = '\n'
+ return f"{self.__name__}(resources=[{new_line.join([r.__repr__() for r in self._resources])}])"
+
+ @classmethod
+ @abstractmethod
+ def _resource_cls(cls) -> Type[Resource]:
+ """Returns the class of the resource that are bundled.
+
+ Returns
+ -------
+ model.resource.Resource
+ """
+ raise NotImplementedError
+
+ @classmethod
+ def from_json(cls, json_string: str) -> ResourceContainer:
+ """Creates a `ResourceContainer` from the given JSON string.
+
+ Parameters
+ ----------
+ json_string: str
+ The JSON string returned from the StreamPipes API.
+
+ Returns
+ -------
+ ResourceContainer
+
+ Raises
+ ------
+ StreamPipesDataModelError
+ If a resource cannot be mapped to the corresponding Python data model.
+ StreamPipesResourceContainerJSONError
+ If JSON response cannot be parsed to a `ResourceContainer`.
+ """
+
+ # deserialize JSON string
+ parsed_json = json.loads(json_string)
+
+ # the ResourceContainer expects a list of items
+ # raise an exception if the response does not be a list
+ if not type(parsed_json) == list:
+ raise StreamPipesResourceContainerJSONError(
+ container=cls.__class__,
+ json_string=json_string
+ )
+ try:
+
+ resource_container = cls(resources=[cls._resource_cls().parse_obj(item) for item in parsed_json])
+ except ValidationError as ve:
+ raise StreamPipesDataModelError(validation_error=ve)
+
+ return resource_container
+
+ def to_dicts(self, use_source_names: bool = False) -> List[Dict]:
+ """Returns the contained resources as list of dictionaries.
+
+ Parameters
+ ----------
+ use_source_names: bool
+ Determines whether the field names are named in Python style (=`False`) or
+ as originally named by StreamPipes (=`True`).
+
+ Returns
+ -------
+ List[Dict]]
+ """
+ return [
+ resource.dict(by_alias=use_source_names) for resource in self._resources
+ ]
+
+ def to_json(self) -> str:
+ """Returns the resource container in the StreamPipes JSON representation.
+
+ Returns
+ -------
+ JSON string
+ """
+
+ return json.dumps(self.to_dicts(use_source_names=True))
+
+ @abstractmethod
+ def to_pandas(self) -> pd.DataFrame:
+ """Returns the resource container in representation of a Pandas Dataframe.
+
+ Returns
+ -------
+ pd.DataFrame
+ """
+ raise NotImplementedError
diff --git a/streampipes-client-python/streampipes_client/model/element/__init__.py b/streampipes-client-python/streampipes_client/model/element/__init__.py
deleted file mode 100644
index a8f3b03aa..000000000
--- a/streampipes-client-python/streampipes_client/model/element/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from .element import Element
diff --git a/streampipes-client-python/streampipes_client/model/exception.py b/streampipes-client-python/streampipes_client/model/exception.py
new file mode 100644
index 000000000..00808216b
--- /dev/null
+++ b/streampipes-client-python/streampipes_client/model/exception.py
@@ -0,0 +1,86 @@
+from abc import abstractmethod
+
+from pydantic import ValidationError
+
+from model.common import BasicModel
+
+__all__ = [
+ "StreamPipesDataModelError",
+ "StreamPipesResourceContainerJSONError"
+]
+
+
+class StreamPipesBaseException(Exception):
+ """Basic class for custom StreamPipes exceptions.
+ """
+
+ @staticmethod
+ @abstractmethod
+ def _generate_error_message(**kwargs):
+ raise NotImplementedError
+
+
+class StreamPipesDataModelError(StreamPipesBaseException):
+ """A custom exception to be raised when a validation error occurs
+ during the parsing of StreamPipes API responses.
+
+ Parameters
+ ----------
+ validation_error: ValidationError
+ The validation error thrown by Pydantic during parsing.
+ """
+ def __init__(self,
+ validation_error: ValidationError,
+ ):
+ super().__init__(
+ self._generate_error_message(
+ model=validation_error.model,
+ error_description=validation_error.json(),
+ )
+ )
+
+ @staticmethod
+ def _generate_error_message(model: BasicModel, error_description: str) -> str:
+ return (
+ f"\nOops, there seems to be a problem with our internal StreamPipes data model.\n"
+ f"This should not occur, but unfortunately did.\n"
+ f"Therefore, it would be great if you could report this problem as an issue at github.com/apache/incubator-streampipes.\n"
+ f"Please don't forget to include the following information:\n\n"
+ f"Affected Model class: {str(model)}\n"
+ f"Validation error log: {error_description}"
+ )
+
+
+class StreamPipesResourceContainerJSONError(StreamPipesBaseException):
+ """A custom exception to be raised when the returned JSON string
+ does not suit to the structure of resource container.
+
+ Parameters
+ ----------
+ container: ResourceContainer
+ The class of the resource container where the invalid data structure was detected.
+ json_string: str
+ The JSON string that has been tried to parse.
+ """
+ def __init__(self,
+ container: "ResourceContainer",
+ json_string: str,
+ ):
+ super().__init__(
+ self._generate_error_message(
+ container=container,
+ json_string=json_string,
+ )
+ )
+
+ @staticmethod
+ def _generate_error_message(container: "ResourceContainer", json_string: str):
+ return (
+ f"\nOops, there seems to be a problem when parsing the response of the StreamPipes API."
+ f"This should not occur, but unfortunately did.\n"
+ f"Therefore, it would be great if you could report this problem as an issue at "
+ f"github.com/apache/incubator-streampipes.\n"
+ f"Please don't forget to include the following information:\n\n"
+ f"Affected container class: {str(container)}\n"
+ f"JSON string: {json_string}"
+ )
diff --git a/streampipes-client-python/streampipes_client/model/element/element.py b/streampipes-client-python/streampipes_client/model/resource/__init__.py
similarity index 85%
rename from streampipes-client-python/streampipes_client/model/element/element.py
rename to streampipes-client-python/streampipes_client/model/resource/__init__.py
index d52f261f7..15b2423ce 100644
--- a/streampipes-client-python/streampipes_client/model/element/element.py
+++ b/streampipes-client-python/streampipes_client/model/resource/__init__.py
@@ -14,10 +14,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from abc import ABC
-from streampipes_client.model.common import BasicModel
+from .resource import Resource
+from .data_lake_measure import DataLakeMeasure
-
-class Element(ABC, BasicModel):
- pass
+__all__ = [
+ "DataLakeMeasure",
+ "Resource"
+]
diff --git a/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py b/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
similarity index 64%
rename from streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
rename to streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
index ca5a89fd9..7d5363724 100644
--- a/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
@@ -18,10 +18,24 @@ from typing import Optional
from pydantic import StrictBool, StrictStr
from streampipes_client.model.common import EventSchema
-from streampipes_client.model.element import Element
+from streampipes_client.model.resource import Resource
+"""
+Implementation of a resource for a data lake measure.
+"""
-class DataLakeMeasure(Element):
+
+__all__ = [
+ "DataLakeMeasure",
+]
+
+class DataLakeMeasure(Resource):
+ """Implementation of a resource for data lake measures.
+ This resource defines the data model used by resource container (`model.container.DataLakeMeasures`).
+ It inherits from Pydantic's BaseModel to get all its superpowers,
+ which are used to parse, validate the API response and to easily switch between
+ the Python representation (both serialized and deserialized) and Java representation (serialized only).
+ """
measure_name: StrictStr
timestamp_field: StrictStr
event_schema: EventSchema
diff --git a/streampipes-client-python/streampipes_client/client/credentials.py b/streampipes-client-python/streampipes_client/model/resource/resource.py
similarity index 51%
rename from streampipes-client-python/streampipes_client/client/credentials.py
rename to streampipes-client-python/streampipes_client/model/resource/resource.py
index de60cb95b..804fb43a1 100644
--- a/streampipes-client-python/streampipes_client/client/credentials.py
+++ b/streampipes-client-python/streampipes_client/model/resource/resource.py
@@ -14,31 +14,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from abc import ABC, abstractmethod
-from typing import Dict, Optional
+"""
+General and abstract implementation for a resource.
+A resource defines the data model that is used by a resource container (`model.container.resourceContainer`).
+"""
+from abc import ABC
-class CredentialProvider(ABC):
- @abstractmethod
- def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
- raise NotImplementedError
+from streampipes_client.model.common import BasicModel
+__all__ = [
+ "Resource",
+]
-class StreamPipesApiKeyCredentials(CredentialProvider):
- def __init__(
- self,
- *,
- username: str,
- api_key: str,
- ):
- self.username = username
- self.api_key = api_key
- def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
-
- if http_headers is None:
- http_headers = {}
-
- http_headers.update({"X-API-User": self.username, "X-API-Key": self.api_key})
-
- return http_headers
+class Resource(ABC, BasicModel):
+ """General and abstract implementation for a resource.
+ A resource defines the data model used by a resource container (`model.container.resourceContainer`).
+ It inherits from Pydantic's BaseModel to get all its superpowers,
+ which are used to parse, validate the API response and to easily switch between
+ the Python representation (both serialized and deserialized) and Java representation (serialized only).
+ """
+ pass
[incubator-streampipes] 06/18: [STREAMPIPES-622] Improve README structure
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 65ee2159163550c81125a71d2f6937e0e0a32626
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Wed Nov 9 22:15:38 2022 +0100
[STREAMPIPES-622] Improve README structure
---
README.md | 35 ++++++++++++++++++++++++-----------
1 file changed, 24 insertions(+), 11 deletions(-)
diff --git a/README.md b/README.md
index bb53b65e1..cbcd705ca 100644
--- a/README.md
+++ b/README.md
@@ -35,7 +35,6 @@
<h3 align="center">Self-Service Data Analytics for the (Industrial) IoT</h3>
<h4 align="center">StreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect
, analyze and explore IoT data streams. </h4>
-</p>
***
@@ -63,11 +62,13 @@ Apache StreamPipes (incubating) makes industrial data analytics easy!
StreamPipes is an end-to-end toolbox for the industrial IoT.
It comes with a rich graphical user interface targeted at non-technical users and provides the following features:
-* Quickly connect >20 industrial data sources and protocols such as OPC-UA, PLCs, MQTT, REST, Kafka and others
+
+* Quickly connect >20 industrial data sources and protocols such as OPC-UA, PLCs, MQTT, REST, Pulsar, Kafka and others
* Create data harmonization and analytics pipelines using > 100 algorithms and many data sinks to forward data to third-party systems
* Use the data explorer to visually explore historical data with many widgets tailored for time-series data.
* A live dashboard to display real-time data from data sources and pipelines, e.g., for shopfloor monitoring.
+
StreamPipes is highly extensible and includes a Java and Python (currently in development phase) SDK to create new
pipeline elements and adapters.
Pipeline elements are standalone microservices that can run anywhere -
@@ -76,6 +77,7 @@ You want to employ your own machine learning model on live data?
Just write your own data processor and make it reusable as a pipeline element.
Besides that, StreamPipes includes features for production deployments:
+
* Assign resources such as pipelines, data streams and dashboards to assets for better organization
* Built-in user and access rights management
* User self-registration and email support
@@ -114,7 +116,15 @@ For a more in-depth manual, read the [installation guide](https://streampipes.ap
## Documentation
-The full documentation, including an overview of pipeline elements, a developer guide and several tutorials is available at [https://streampipes.apache.org/docs/docs/user-guide-introduction.html](https://streampipes.apache.org/docs/docs/user-guide-introduction.html)
+The full documentationis available [here](https://streampipes.apache.org/docs/index.html).
+
+Quick Links:
+
+* [Installation](https://streampipes.apache.org/docs/docs/try-installation.html)
+* [Create adapters](https://streampipes.apache.org/docs/docs/use-connect.html)
+* [Create pipelines](https://streampipes.apache.org/docs/docs/use-pipeline-editor.html)
+* [Pipeline elements](https://streampipes.apache.org/pipeline-elements.html)
+* [Write you own pipeline elements](https://streampipes.apache.org/docs/docs/extend-archetypes.html)
## Building StreamPipes
@@ -149,32 +159,35 @@ To start StreamPipes, run ``docker-compose up --build -d`` from the root directo
You can also use the installer or CLI as described in the ``Installation`` section.
## Pipeline Elements
-StreamPipes includes a repository of extensions for
+StreamPipes includes a repository of extensions for adapters and pipeline elements:
* **Connect adapters** for a variety of IoT data sources as well as
* **Data Processors** and **Data Sinks** as ready-to-use pipeline elements.
-A description of the standard elements can be found in [streampipes-extensions](https://github.com/apache/incubator-streampipes/tree/dev/streampipes-extensions).
+The source code of all included pipeline elements and adapters can be found [here](https://github.com/apache/incubator-streampipes/tree/dev/streampipes-extensions).
## Extending StreamPipes
-You can easily add your own data streams, processors or sinks. A [Java-based SDK](https://streampipes.apache.org/docs/docs/dev-guide-tutorial-processors) and several [run-time wrappers](https://streampipes.apache.org/docs/docs/dev-guide-architecture) for popular streaming frameworks such as Apache Flink and Apache
-Kafka Streams (and also plain Java programs) can be used to integrate your existing processing logic into StreamPipes. Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.
+You can easily add your own data streams, processors or sinks. A [Java-based SDK](https://streampipes.apache.org/docs/docs/dev-guide-tutorial-processors) can be used to integrate your existing processing logic into StreamPipes.
+Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.
-Check our developer guide at [https://streampipes.apache.org/docs/docs/dev-guide-introduction](https://streampipes.apache.org/docs/docs/dev-guide-introduction).
+👉 Check our developer guide at [https://streampipes.apache.org/docs/docs/dev-guide-introduction](https://streampipes.apache.org/docs/docs/dev-guide-introduction).
## Bugs and Feature Requests
If you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue in our Jira:
-[https://issues.apache.org/jira/projects/STREAMPIPES](https://issues.apache.org/jira/projects/STREAMPIPES)
+
+👉 [https://issues.apache.org/jira/projects/STREAMPIPES](https://issues.apache.org/jira/projects/STREAMPIPES)
## Get help
If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our
community channels:
-- [Mailing Lists](https://streampipes.apache.org/mailinglists.html)
+👉 [Mailing Lists](https://streampipes.apache.org/mailinglists.html)
+
+Or directly subscribe to [users-subscribe@streampipes.apache.org](mailto:users-subscribe@streampipes.apache.org)!
-And don't forget to follow us on [Twitter](https://twitter.com/streampipes)!
+👉 And don't forget to follow us on [Twitter](https://twitter.com/streampipes)!
## Contribute
[incubator-streampipes] 09/18: [STREAMPIPES-622] Fix some links in README file
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 52ba7ee107c7659d3f3887d74750e2d663d8bbec
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Wed Nov 9 22:24:37 2022 +0100
[STREAMPIPES-622] Fix some links in README file
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 06e681529..a593daeb0 100644
--- a/README.md
+++ b/README.md
@@ -167,10 +167,10 @@ The source code of all included pipeline elements and adapters can be found [her
## Extending StreamPipes
-You can easily add your own data streams, processors or sinks. A [Java-based SDK](https://streampipes.apache.org/docs/docs/dev-guide-tutorial-processors) can be used to integrate your existing processing logic into StreamPipes.
+You can easily add your own data streams, processors or sinks. A [Java-based SDK](https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html) can be used to integrate your existing processing logic into StreamPipes.
Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.
-👉 Check our developer guide at [https://streampipes.apache.org/docs/docs/dev-guide-introduction](https://streampipes.apache.org/docs/docs/dev-guide-introduction).
+👉 Check our developer guide at [https://streampipes.apache.org/docs/docs/extend-setup.html](https://streampipes.apache.org/docs/docs/extend-setup.html).
## Bugs and Feature Requests
[incubator-streampipes] 03/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 5d240655398bb250b5565aeddfc26d97d84cff59
Author: bossenti <bo...@posteo.de>
AuthorDate: Fri Nov 4 19:19:07 2022 +0100
tmp
---
streampipes-client-python/githooks/pre-commit | 10 ++
.../streampipesclient/client/__init__.py | 1 -
.../streampipesclient/client/client.py | 67 +++++++++++--
.../{client => endpoints}/__init__.py | 6 +-
.../endpoints/data_lake_measure.py | 18 ++++
.../streampipesclient/endpoints/endpoint.py | 104 +++++++++++++++++++++
.../{client => model}/__init__.py | 6 +-
.../streampipesclient/model/common.py | 85 +++++++++++++++++
.../{client => model/container}/__init__.py | 4 +-
.../model/container/data_lake_measures.py | 14 +++
.../model/container/model_container.py | 66 +++++++++++++
.../{client => model/element}/__init__.py | 4 +-
.../element/data_lake_measure.py} | 22 +++--
.../{client/config.py => model/element/element.py} | 17 +---
14 files changed, 378 insertions(+), 46 deletions(-)
diff --git a/streampipes-client-python/githooks/pre-commit b/streampipes-client-python/githooks/pre-commit
new file mode 100644
index 000000000..025fc6455
--- /dev/null
+++ b/streampipes-client-python/githooks/pre-commit
@@ -0,0 +1,10 @@
+#!/bin/sh
+set -e
+
+CHANGED_FILES="$(git --no-pager diff --name-status --no-color --cached --diff-filter=d|awk '$1 !~ /R/ && ( $2 ~ /\.py|\.pyi/ ) {print $2}; $1 ~ /R/ && ( $3 ~ /\.py|\.pyi/ ) {print $3}')"
+
+if [ -z "$CHANGED_FILES" ]; then
+ echo "No Python staged files."
+ exit 0
+fi;
+
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/client/__init__.py
index 56ed17e8a..a323247a5 100644
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ b/streampipes-client-python/streampipesclient/client/__init__.py
@@ -16,5 +16,4 @@
#
from .client import StreamPipesClient
-from .config import StreamPipesClientConfig
from .credentials import StreamPipesApiKeyCredentials
diff --git a/streampipes-client-python/streampipesclient/client/client.py b/streampipes-client-python/streampipesclient/client/client.py
index f2855bbeb..3df75869b 100644
--- a/streampipes-client-python/streampipesclient/client/client.py
+++ b/streampipes-client-python/streampipesclient/client/client.py
@@ -14,17 +14,72 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+import logging
+import sys
+from dataclasses import dataclass
+from typing import Dict, Optional, Union, Type
+
+import requests
+
+from streampipesclient.client.credentials import CredentialProvider
+from streampipesclient.endpoints.data_lake_measure import DataLakeMeasureEndpoint
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class StreamPipesClientConfig:
+ """
+
+ """
+
+ credential_provider: CredentialProvider
+ host_address: str
+ https_disabled: Optional[bool] = False
+ port: Optional[int] = 80
+
class StreamPipesClient:
- def __init__(self):
+ def __init__(self,
+ *,
+ client_config: StreamPipesClientConfig,
+ logging_level: int = logging.INFO,
+ ):
+ self.client_config = client_config
+
+ self.request_session = requests.Session()
+ self.request_session.headers.update(self.http_headers)
+
+ self._set_up_logging(logging_level=logging_level)
+
+ # endpoints
+ self.dataLakeMeasureApi = DataLakeMeasureEndpoint(parent_client=self)
+
+ @staticmethod
+ def _set_up_logging(*, logging_level: int) -> None:
+ logging.basicConfig(
+ level=logging_level,
+ stream=sys.stdout,
+ format="%(asctime)s - %(name)s - [%(levelname)s] - [%(filename)s:%(lineno)d] [%(funcName)s] - %(message)s",
+ )
+
+ logger.info(f"Logging successfully initialized with logging level {logging.getLevelName(logging_level)}.")
- pass
+ @classmethod
+ def create(cls, *, client_config: StreamPipesClientConfig, logging_level: int = logging.INFO):
+ return cls(client_config=client_config, logging_level=logging_level)
@property
- def http_headers(self):
- pass
+ def http_headers(self) -> Dict[str, str]:
+ return self.client_config.credential_provider.make_headers(
+ {
+ "Application": "application/json"
+ }
+ )
@property
- def base_api_path(self):
- pass
\ No newline at end of file
+ def base_api_path(self) -> str:
+ return f"{'http://' if self.client_config.https_disabled else 'https://'}" \
+ f"{self.client_config.host_address}:" \
+ f"{self.client_config.port}/streampipes-backend/"
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/endpoints/__init__.py
similarity index 84%
copy from streampipes-client-python/streampipesclient/client/__init__.py
copy to streampipes-client-python/streampipesclient/endpoints/__init__.py
index 56ed17e8a..ecb1860df 100644
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ b/streampipes-client-python/streampipesclient/endpoints/__init__.py
@@ -13,8 +13,4 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
-
-from .client import StreamPipesClient
-from .config import StreamPipesClientConfig
-from .credentials import StreamPipesApiKeyCredentials
+#
\ No newline at end of file
diff --git a/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py b/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py
new file mode 100644
index 000000000..09c5e7fab
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py
@@ -0,0 +1,18 @@
+from typing import List
+
+from streampipesclient.endpoints.endpoint import APIEndpoint
+from streampipesclient.model.container import ModelContainer
+from streampipesclient.model.container.data_lake_measures import DataLakeMeasures
+from streampipesclient.model.element import Element
+from streampipesclient.model.element.data_lake_measure import DataLakeMeasure
+
+
+class DataLakeMeasureEndpoint(APIEndpoint):
+
+ @classmethod
+ def _container_cls(cls):
+ return DataLakeMeasures
+
+ @property
+ def _relative_api_path(self):
+ return "api", "v4", "datalake", "measurements"
diff --git a/streampipes-client-python/streampipesclient/endpoints/endpoint.py b/streampipes-client-python/streampipesclient/endpoints/endpoint.py
new file mode 100644
index 000000000..b0a5c4693
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/endpoints/endpoint.py
@@ -0,0 +1,104 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from http import HTTPStatus
+import logging
+from abc import ABC, abstractmethod
+from typing import Tuple, Type, Callable
+
+import requests
+from requests import Response
+from requests.exceptions import HTTPError
+
+from streampipesclient.model.container import ModelContainer
+from streampipesclient.model.element import Element
+
+logger = logging.getLogger(__name__)
+
+status_code_to_log_message = {
+ 401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
+ "Please check your user name and/or password to be correct.",
+ 403: "\nThere seems to be an issue with the accees rights of the given user and the ressource you queried.\n"
+ "Apparently, this user is not allowd to query the resource.\n"
+ "Please check the user's permissions or contact your StreamPipes admin.",
+ **dict.fromkeys(
+ [404, 405],
+ "\nOops, there seems to be an issue with the Python Client calling the API inappropriately.\n"
+ "This should not happen, but unfortunately did.\n"
+ "If you don't mind, it would be awesome to let us know by creating an issue at github.com/apache/incubator-streampipes.\n"
+ "Please paste the following information to the issue description:\n\n")
+}
+
+
+class APIEndpoint(ABC):
+
+ def __init__(self, parent_client: "StreamPipesClient"):
+ self._parent_client = parent_client
+
+ @property
+ @abstractmethod
+ def _relative_api_path(self) -> Tuple[str]:
+ raise NotImplementedError
+
+ @classmethod
+ @abstractmethod
+ def _container_cls(cls) -> Type[ModelContainer]:
+ raise NotImplementedError
+
+ def _make_request(self,
+ *,
+ request_method: Callable[..., Response],
+ url: str,
+ ) -> Response:
+
+ response = request_method(url=url)
+
+ try:
+ response.raise_for_status()
+ except HTTPError as err:
+
+ status_code = err.response.status_code
+
+ log_message = status_code_to_log_message[err.response.status_code]
+
+ if status_code in [HTTPStatus.METHOD_NOT_ALLOWED.numerator, HTTPStatus.NOT_FOUND.numerator]:
+ log_message += f"url: {err.response.url}\nstatus code: {err.response.status_code}"
+
+ logger.debug(err.response.text)
+ raise HTTPError(log_message) from err
+
+ else:
+ logger.debug("Successfully retrieved resources from %s.", url)
+ logger.info("Successfully retrieved all resources.")
+
+ return response
+
+ def create_api_path(self) -> str:
+ return f"{self._parent_client.base_api_path}{'/'.join(api_path for api_path in self._relative_api_path)}"
+
+ def all(self) -> ModelContainer:
+
+ response = self._make_request(
+ request_method=self._parent_client.request_session.get,
+ url=self.create_api_path()
+ )
+ return self._container_cls().from_json(json_string=response.text)
+
+ def get(self, *, identifier: str) -> Element:
+
+ # equals to download
+ # needs further considerations
+ pass
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/model/__init__.py
similarity index 84%
copy from streampipes-client-python/streampipesclient/client/__init__.py
copy to streampipes-client-python/streampipesclient/model/__init__.py
index 56ed17e8a..ecb1860df 100644
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ b/streampipes-client-python/streampipesclient/model/__init__.py
@@ -13,8 +13,4 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
-
-from .client import StreamPipesClient
-from .config import StreamPipesClientConfig
-from .credentials import StreamPipesApiKeyCredentials
+#
\ No newline at end of file
diff --git a/streampipes-client-python/streampipesclient/model/common.py b/streampipes-client-python/streampipesclient/model/common.py
new file mode 100644
index 000000000..b1994372f
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/model/common.py
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import List, Optional
+
+from pydantic import BaseModel, StrictStr, StrictInt, StrictBool, ValidationError
+
+
+def snake_to_camel_case(snake_case_string: str) -> str:
+ """
+ Converts a string in snake_case format to camelCase style.
+ """
+
+ tokens = snake_case_string.split("_")
+
+ return tokens[0] + "".join(t.title() for t in tokens[1:])
+
+
+class BasicModel(BaseModel):
+ element_id: Optional[StrictStr]
+
+ class Config:
+ alias_generator = snake_to_camel_case
+
+
+class EventPropertyQualityDefinition(BasicModel):
+ pass
+
+
+class EventPropertyQualityRequirement(BasicModel):
+ minimum_property_quality: Optional[EventPropertyQualityDefinition]
+ maximum_property_quality: Optional[EventPropertyQualityDefinition]
+
+
+class EventProperty(BasicModel):
+ label: StrictStr
+ description: StrictStr
+ runtime_name: StrictStr
+ required: StrictBool
+ domain_properties: List[StrictStr]
+ event_property_qualities: List[EventPropertyQualityDefinition]
+ requires_event_property_qualities: List[EventPropertyQualityRequirement]
+ property_scope: Optional[StrictStr]
+ index: StrictInt
+ runtime_id: Optional[StrictStr]
+ runtime_type: Optional[StrictStr]
+ measurement_unit: Optional[StrictStr]
+ value_specification: Optional[StrictStr]
+
+
+class EventSchema(BasicModel):
+ event_properties: List[EventProperty]
+
+
+class StreamPipesDataModelError(RuntimeError):
+
+ def __init__(self, validation_error: ValidationError):
+ super().__init__(
+ self._generate_error_message(model=validation_error.model,
+ error_description=validation_error.json(),
+ )
+ )
+
+ @staticmethod
+ def _generate_error_message(*, model: BasicModel, error_description: str) -> str:
+ return f"\nOops, there seems to be a problem with our internal StreamPipes data model.\n" \
+ f"This should not occur, but unfortunately did.\n" \
+ f"Therefore, it would be great if you could report this problem as an issue at github.com/apache/incubator-streampipes.\n" \
+ f"Please don't forget to include the following information:\n\n" \
+ f"Affected Model class: {str(model)}\n" \
+ f"Validation error log: {error_description}"
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/model/container/__init__.py
similarity index 85%
copy from streampipes-client-python/streampipesclient/client/__init__.py
copy to streampipes-client-python/streampipesclient/model/container/__init__.py
index 56ed17e8a..9d8d2b197 100644
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ b/streampipes-client-python/streampipesclient/model/container/__init__.py
@@ -15,6 +15,4 @@
# limitations under the License.
#
-from .client import StreamPipesClient
-from .config import StreamPipesClientConfig
-from .credentials import StreamPipesApiKeyCredentials
+from .model_container import ModelContainer
diff --git a/streampipes-client-python/streampipesclient/model/container/data_lake_measures.py b/streampipes-client-python/streampipesclient/model/container/data_lake_measures.py
new file mode 100644
index 000000000..11005510a
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/model/container/data_lake_measures.py
@@ -0,0 +1,14 @@
+import pandas as pd
+
+from streampipesclient.model.container import ModelContainer
+from streampipesclient.model.element.data_lake_measure import DataLakeMeasure
+
+
+class DataLakeMeasures(ModelContainer):
+
+ @classmethod
+ def _element_cls(cls):
+ return DataLakeMeasure
+
+ def to_pandas(self) -> pd.DataFrame:
+ pass
\ No newline at end of file
diff --git a/streampipes-client-python/streampipesclient/model/container/model_container.py b/streampipes-client-python/streampipesclient/model/container/model_container.py
new file mode 100644
index 000000000..666f06ac9
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/model/container/model_container.py
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import json
+
+import pandas as pd
+
+from abc import ABC, abstractmethod
+from typing import List, Type
+
+from pydantic import ValidationError
+
+from streampipesclient.model.common import StreamPipesDataModelError
+from streampipesclient.model.element import Element
+
+
+class ModelContainer(ABC):
+
+ def __init__(self, elements: List[Element]):
+ self._elements = elements
+
+ def __getitem__(self, position: int) -> Element:
+ return self._elements[position]
+
+ def __len__(self) -> int:
+ return len(self._elements)
+
+ @classmethod
+ def from_json(cls, json_string: str) -> "ModelContainer":
+
+ data = json.loads(json_string)
+
+ if not type(data) == list:
+ raise RuntimeError
+
+ try:
+
+ model_container = cls(
+ elements=[cls._element_cls().parse_obj(list_item) for list_item in data]
+ )
+ except ValidationError as ve:
+ raise StreamPipesDataModelError(validation_error=ve)
+
+ return model_container
+
+ @abstractmethod
+ def to_pandas(self) -> pd.DataFrame:
+ raise NotImplementedError
+
+ @classmethod
+ @abstractmethod
+ def _element_cls(cls) -> Type[Element]:
+ raise NotImplementedError
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/model/element/__init__.py
similarity index 85%
copy from streampipes-client-python/streampipesclient/client/__init__.py
copy to streampipes-client-python/streampipesclient/model/element/__init__.py
index 56ed17e8a..f4b8a3ece 100644
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ b/streampipes-client-python/streampipesclient/model/element/__init__.py
@@ -15,6 +15,4 @@
# limitations under the License.
#
-from .client import StreamPipesClient
-from .config import StreamPipesClientConfig
-from .credentials import StreamPipesApiKeyCredentials
+from .element import Element
\ No newline at end of file
diff --git a/streampipes-client-python/streampipesclient/client/client.py b/streampipes-client-python/streampipesclient/model/element/data_lake_measure.py
similarity index 64%
copy from streampipes-client-python/streampipesclient/client/client.py
copy to streampipes-client-python/streampipesclient/model/element/data_lake_measure.py
index f2855bbeb..7601bc115 100644
--- a/streampipes-client-python/streampipesclient/client/client.py
+++ b/streampipes-client-python/streampipesclient/model/element/data_lake_measure.py
@@ -14,17 +14,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+from typing import Optional
-class StreamPipesClient:
+from pydantic import StrictStr, StrictBool
- def __init__(self):
+from streampipesclient.model.common import EventSchema
+from streampipesclient.model.element import Element
- pass
- @property
- def http_headers(self):
- pass
-
- @property
- def base_api_path(self):
- pass
\ No newline at end of file
+class DataLakeMeasure(Element):
+ measure_name: StrictStr
+ timestamp_field: StrictStr
+ event_schema: EventSchema
+ pipeline_id: Optional[StrictStr]
+ pipeline_name: Optional[StrictStr]
+ pipeline_is_running: StrictBool
+ schema_version: StrictStr
diff --git a/streampipes-client-python/streampipesclient/client/config.py b/streampipes-client-python/streampipesclient/model/element/element.py
similarity index 69%
rename from streampipes-client-python/streampipesclient/client/config.py
rename to streampipes-client-python/streampipesclient/model/element/element.py
index b1f62fe24..bff745778 100644
--- a/streampipes-client-python/streampipesclient/client/config.py
+++ b/streampipes-client-python/streampipesclient/model/element/element.py
@@ -14,19 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from dataclasses import dataclass
-from typing import Optional, Type
+from abc import ABC
-from streampipesclient.client.credentials import CredentialProvider
+from streampipesclient.model.common import BasicModel
-@dataclass
-class StreamPipesClientConfig:
- """
-
- """
-
- credential_provider: Type[CredentialProvider]
- host_address: str
- https_disabled: Optional[bool] = False
- port: Optional[int] = 80
+class Element(ABC, BasicModel):
+ pass
[incubator-streampipes] 12/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 49830cae572016474a6dc56d07101bfffcc1d427
Author: bossenti <bo...@posteo.de>
AuthorDate: Thu Nov 10 21:31:08 2022 +0100
tmp
---
streampipes-client-python/.gitignore | 3 ++-
streampipes-client-python/README.md | 2 ++
streampipes-client-python/githooks/pre-commit | 10 ----------
streampipes-client-python/setup.py | 4 +++-
.../endpoints => streampipes_client}/__init__.py | 0
.../streampipes_client/__version__.py | 2 +-
.../streampipes_client/client/__init__.py | 5 ++++-
.../client/client.py | 6 +++---
.../client/credentials.py | 0
.../streampipes_client/endpoint/data_lake_measure.py | 13 +++++++++++++
.../endpoint}/endpoint.py | 5 ++---
.../model/common.py | 0
.../model/container/__init__.py | 0
.../model/container/data_lake_measures.py | 4 ++--
.../model/container/model_container.py | 4 ++--
.../model/element/__init__.py | 0
.../model/element/data_lake_measure.py | 4 ++--
.../model/element/element.py | 2 +-
.../streampipesclient/__init__.py | 18 ------------------
.../streampipesclient/client/__init__.py | 19 -------------------
.../streampipesclient/endpoints/data_lake_measure.py | 18 ------------------
.../streampipesclient/model/__init__.py | 16 ----------------
streampipes-client-python/tests/__init__.py | 0
23 files changed, 37 insertions(+), 98 deletions(-)
diff --git a/streampipes-client-python/.gitignore b/streampipes-client-python/.gitignore
index 0cd13ceab..3de8f73f6 100644
--- a/streampipes-client-python/.gitignore
+++ b/streampipes-client-python/.gitignore
@@ -186,7 +186,6 @@ cover/
.pybuilder/
target/
-# Jupyter Notebook
# IPython
@@ -216,6 +215,8 @@ venv/
ENV/
env.bak/
venv.bak/
+build
+Scripts
# Spyder project settings
.spyderproject
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index 4fbb650c5..c4164d196 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -43,6 +43,8 @@ and the amazing universe of data analytics libraries in Python.
<br>
**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
+<br>
+
### Get ready for development
1) Set up your Python environment
diff --git a/streampipes-client-python/githooks/pre-commit b/streampipes-client-python/githooks/pre-commit
deleted file mode 100644
index 025fc6455..000000000
--- a/streampipes-client-python/githooks/pre-commit
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/sh
-set -e
-
-CHANGED_FILES="$(git --no-pager diff --name-status --no-color --cached --diff-filter=d|awk '$1 !~ /R/ && ( $2 ~ /\.py|\.pyi/ ) {print $2}; $1 ~ /R/ && ( $3 ~ /\.py|\.pyi/ ) {print $3}')"
-
-if [ -z "$CHANGED_FILES" ]; then
- echo "No Python staged files."
- exit 0
-fi;
-
diff --git a/streampipes-client-python/setup.py b/streampipes-client-python/setup.py
index 4c1a74fff..4914564f8 100644
--- a/streampipes-client-python/setup.py
+++ b/streampipes-client-python/setup.py
@@ -31,6 +31,9 @@ REQUIRES_PYTHON = ">=3.8.0"
# Package requirements.
base_packages = [
+ "pandas>=1.5.1",
+ "pydantic>=1.10.2",
+ "requests>=2.28.1"
]
dev_packages = base_packages + [
@@ -79,7 +82,6 @@ setuptools.setup(
"test": dev_packages,
"docs": docs_packages,
"all": dev_packages + docs_packages,
- ":python_version == '3.8'": ["dataclasses"], # todo check if required
},
include_package_data=True,
license="Apache License 2.0",
diff --git a/streampipes-client-python/streampipesclient/endpoints/__init__.py b/streampipes-client-python/streampipes_client/__init__.py
similarity index 100%
rename from streampipes-client-python/streampipesclient/endpoints/__init__.py
rename to streampipes-client-python/streampipes_client/__init__.py
diff --git a/streampipes-client-python/streampipes_client/__version__.py b/streampipes-client-python/streampipes_client/__version__.py
index 3496c1813..492084802 100644
--- a/streampipes-client-python/streampipes_client/__version__.py
+++ b/streampipes-client-python/streampipes_client/__version__.py
@@ -15,6 +15,6 @@
# limitations under the License.
#
-VERSION = (0, 0, 1)
+VERSION = (0, 71, 0)
__version__ = ".".join(map(str, VERSION)) # noqa: F401
diff --git a/streampipes-client-python/streampipes_client/client/__init__.py b/streampipes-client-python/streampipes_client/client/__init__.py
index ecb1860df..a323247a5 100644
--- a/streampipes-client-python/streampipes_client/client/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/__init__.py
@@ -13,4 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
\ No newline at end of file
+#
+
+from .client import StreamPipesClient
+from .credentials import StreamPipesApiKeyCredentials
diff --git a/streampipes-client-python/streampipesclient/client/client.py b/streampipes-client-python/streampipes_client/client/client.py
similarity index 93%
rename from streampipes-client-python/streampipesclient/client/client.py
rename to streampipes-client-python/streampipes_client/client/client.py
index 3df75869b..f7948b7c0 100644
--- a/streampipes-client-python/streampipesclient/client/client.py
+++ b/streampipes-client-python/streampipes_client/client/client.py
@@ -17,12 +17,12 @@
import logging
import sys
from dataclasses import dataclass
-from typing import Dict, Optional, Union, Type
+from typing import Dict, Optional
import requests
-from streampipesclient.client.credentials import CredentialProvider
-from streampipesclient.endpoints.data_lake_measure import DataLakeMeasureEndpoint
+from streampipes_client.client.credentials import CredentialProvider
+from streampipesclient.endpoints import DataLakeMeasureEndpoint
logger = logging.getLogger(__name__)
diff --git a/streampipes-client-python/streampipesclient/client/credentials.py b/streampipes-client-python/streampipes_client/client/credentials.py
similarity index 100%
rename from streampipes-client-python/streampipesclient/client/credentials.py
rename to streampipes-client-python/streampipes_client/client/credentials.py
diff --git a/streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py b/streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py
new file mode 100644
index 000000000..6ee4c8403
--- /dev/null
+++ b/streampipes-client-python/streampipes_client/endpoint/data_lake_measure.py
@@ -0,0 +1,13 @@
+from streampipes_client.endpoint.endpoint import APIEndpoint
+from streampipes_client.model.container.data_lake_measures import DataLakeMeasures
+
+
+class DataLakeMeasureEndpoint(APIEndpoint):
+
+ @classmethod
+ def _container_cls(cls):
+ return DataLakeMeasures
+
+ @property
+ def _relative_api_path(self):
+ return "api", "v4", "datalake", "measurements"
diff --git a/streampipes-client-python/streampipesclient/endpoints/endpoint.py b/streampipes-client-python/streampipes_client/endpoint/endpoint.py
similarity index 96%
rename from streampipes-client-python/streampipesclient/endpoints/endpoint.py
rename to streampipes-client-python/streampipes_client/endpoint/endpoint.py
index b0a5c4693..4bc54df19 100644
--- a/streampipes-client-python/streampipesclient/endpoints/endpoint.py
+++ b/streampipes-client-python/streampipes_client/endpoint/endpoint.py
@@ -19,12 +19,11 @@ import logging
from abc import ABC, abstractmethod
from typing import Tuple, Type, Callable
-import requests
from requests import Response
from requests.exceptions import HTTPError
-from streampipesclient.model.container import ModelContainer
-from streampipesclient.model.element import Element
+from streampipes_client.model.container import ModelContainer
+from streampipes_client.model.element import Element
logger = logging.getLogger(__name__)
diff --git a/streampipes-client-python/streampipesclient/model/common.py b/streampipes-client-python/streampipes_client/model/common.py
similarity index 100%
rename from streampipes-client-python/streampipesclient/model/common.py
rename to streampipes-client-python/streampipes_client/model/common.py
diff --git a/streampipes-client-python/streampipesclient/model/container/__init__.py b/streampipes-client-python/streampipes_client/model/container/__init__.py
similarity index 100%
rename from streampipes-client-python/streampipesclient/model/container/__init__.py
rename to streampipes-client-python/streampipes_client/model/container/__init__.py
diff --git a/streampipes-client-python/streampipesclient/model/container/data_lake_measures.py b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
similarity index 54%
rename from streampipes-client-python/streampipesclient/model/container/data_lake_measures.py
rename to streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
index 11005510a..758a57a8b 100644
--- a/streampipes-client-python/streampipesclient/model/container/data_lake_measures.py
+++ b/streampipes-client-python/streampipes_client/model/container/data_lake_measures.py
@@ -1,7 +1,7 @@
import pandas as pd
-from streampipesclient.model.container import ModelContainer
-from streampipesclient.model.element.data_lake_measure import DataLakeMeasure
+from streampipes_client.model.container import ModelContainer
+from streampipes_client.model.element.data_lake_measure import DataLakeMeasure
class DataLakeMeasures(ModelContainer):
diff --git a/streampipes-client-python/streampipesclient/model/container/model_container.py b/streampipes-client-python/streampipes_client/model/container/model_container.py
similarity index 93%
rename from streampipes-client-python/streampipesclient/model/container/model_container.py
rename to streampipes-client-python/streampipes_client/model/container/model_container.py
index 666f06ac9..072e590ff 100644
--- a/streampipes-client-python/streampipesclient/model/container/model_container.py
+++ b/streampipes-client-python/streampipes_client/model/container/model_container.py
@@ -23,8 +23,8 @@ from typing import List, Type
from pydantic import ValidationError
-from streampipesclient.model.common import StreamPipesDataModelError
-from streampipesclient.model.element import Element
+from streampipes_client.model.common import StreamPipesDataModelError
+from streampipes_client.model.element import Element
class ModelContainer(ABC):
diff --git a/streampipes-client-python/streampipesclient/model/element/__init__.py b/streampipes-client-python/streampipes_client/model/element/__init__.py
similarity index 100%
rename from streampipes-client-python/streampipesclient/model/element/__init__.py
rename to streampipes-client-python/streampipes_client/model/element/__init__.py
diff --git a/streampipes-client-python/streampipesclient/model/element/data_lake_measure.py b/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
similarity index 91%
rename from streampipes-client-python/streampipesclient/model/element/data_lake_measure.py
rename to streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
index 7601bc115..444978d3e 100644
--- a/streampipes-client-python/streampipesclient/model/element/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/model/element/data_lake_measure.py
@@ -18,8 +18,8 @@ from typing import Optional
from pydantic import StrictStr, StrictBool
-from streampipesclient.model.common import EventSchema
-from streampipesclient.model.element import Element
+from streampipes_client.model.common import EventSchema
+from streampipes_client.model.element import Element
class DataLakeMeasure(Element):
diff --git a/streampipes-client-python/streampipesclient/model/element/element.py b/streampipes-client-python/streampipes_client/model/element/element.py
similarity index 93%
rename from streampipes-client-python/streampipesclient/model/element/element.py
rename to streampipes-client-python/streampipes_client/model/element/element.py
index bff745778..d52f261f7 100644
--- a/streampipes-client-python/streampipesclient/model/element/element.py
+++ b/streampipes-client-python/streampipes_client/model/element/element.py
@@ -16,7 +16,7 @@
#
from abc import ABC
-from streampipesclient.model.common import BasicModel
+from streampipes_client.model.common import BasicModel
class Element(ABC, BasicModel):
diff --git a/streampipes-client-python/streampipesclient/__init__.py b/streampipes-client-python/streampipesclient/__init__.py
deleted file mode 100644
index 43a8ab2c8..000000000
--- a/streampipes-client-python/streampipesclient/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from .client import StreamPipesClient
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/client/__init__.py
deleted file mode 100644
index a323247a5..000000000
--- a/streampipes-client-python/streampipesclient/client/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from .client import StreamPipesClient
-from .credentials import StreamPipesApiKeyCredentials
diff --git a/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py b/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py
deleted file mode 100644
index 09c5e7fab..000000000
--- a/streampipes-client-python/streampipesclient/endpoints/data_lake_measure.py
+++ /dev/null
@@ -1,18 +0,0 @@
-from typing import List
-
-from streampipesclient.endpoints.endpoint import APIEndpoint
-from streampipesclient.model.container import ModelContainer
-from streampipesclient.model.container.data_lake_measures import DataLakeMeasures
-from streampipesclient.model.element import Element
-from streampipesclient.model.element.data_lake_measure import DataLakeMeasure
-
-
-class DataLakeMeasureEndpoint(APIEndpoint):
-
- @classmethod
- def _container_cls(cls):
- return DataLakeMeasures
-
- @property
- def _relative_api_path(self):
- return "api", "v4", "datalake", "measurements"
diff --git a/streampipes-client-python/streampipesclient/model/__init__.py b/streampipes-client-python/streampipesclient/model/__init__.py
deleted file mode 100644
index ecb1860df..000000000
--- a/streampipes-client-python/streampipesclient/model/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
\ No newline at end of file
diff --git a/streampipes-client-python/tests/__init__.py b/streampipes-client-python/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
[incubator-streampipes] 16/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 79354a97fcc5e08af9e2cd4166ae79321ec7d369
Author: bossenti <bo...@posteo.de>
AuthorDate: Sat Nov 12 09:39:55 2022 +0100
tmp
---
streampipes-client-python/DEVELOPMENT.md | 20 ++++++++++++++++++--
.../streampipes_client/client/endpoint/__init__.py | 5 ++++-
.../streampipes_client/client/endpoint/endpoint.py | 8 +++++---
3 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/streampipes-client-python/DEVELOPMENT.md b/streampipes-client-python/DEVELOPMENT.md
index 27bb7d1f3..9bda1f1e4 100644
--- a/streampipes-client-python/DEVELOPMENT.md
+++ b/streampipes-client-python/DEVELOPMENT.md
@@ -28,7 +28,7 @@ Create a virtual Python environment with a tool of your choice, here are some ex
As a next step, install all required dependencies for the development, e.g. with pip:
```
-pip install .[dev] # or alternatively: pip install .[docs]
+pip install .[dev] # or alternatively: pip install .[docs] to include dependencies for building the docs as well
```
<br>
@@ -45,8 +45,24 @@ The definition of the pre-commit hook can be found in [.pre-commit-config.yaml](
### Conventions :clap:
Below we list some conventions that we have agreed on for creating the StreamPipes Python client.
-Please follow them when you plan to contribute to this project.
+Please comply to them when you plan to contribute to this project.
If you have any other suggestions or would like to discuss them, we would be happy to hear from you on our mailing list [dev@streampipes.apache.org](mailto:dev@streampipes.apache.org).
1) **Use `numpy` style for Python docstrings** :page_facing_up: <br>
Please stick to the `numpy` style when writing docstrings, as we require this for generating our documentation.
+
+
+2) **Provide tests** :heavy_check_mark: <br>
+We are aiming for broad test coverage for the Python client and
+have therefore set a requirement of at least 75% unit test coverage.
+Therefore, please remember to write (unit) tests already during development.
+If you have problems with writing tests, don't hesitate to ask us for help directly in the PR or
+even before that via our mailing list (see above).
+
+
+<!---
+TODO: replace link to java file by link to documentation
+--->
+3) **Build a similar API as the Java client provides** :arrows_clockwise: <br>
+Whenever possible, please try to develop the API of the Python client the same as the [Java client](../streampipes-client/src/main/java/org/apache/streampipes/client/StreamPipesClient.java).
+By doing so, we would like to provide a consistent developer experience and the basis for automated testing in the future.
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/__init__.py b/streampipes-client-python/streampipes_client/client/endpoint/__init__.py
index 75f4dca5d..25cdf4113 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/__init__.py
@@ -21,4 +21,7 @@ One endpoint aims to handle the whole interaction with a central endpoint of the
from .data_lake_measure import DataLakeMeasureEndpoint
from .endpoint import APIEndpoint
-__all__ = ["APIEndpoint", "DataLakeMeasureEndpoint"]
+__all__ = [
+ "APIEndpoint",
+ "DataLakeMeasureEndpoint",
+]
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index a113501b7..b4c84b37e 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -31,12 +31,14 @@ from requests.exceptions import BaseHTTPError, HTTPError
from streampipes_client.model.container import ModelContainer
from streampipes_client.model.element import Element
-__all__ = ["APIEndpoint"]
+__all__ = [
+ "APIEndpoint",
+]
logger = logging.getLogger(__name__)
# define custom logging messages for some specific HTTP status
-error_code_to_message = {
+_error_code_to_message = {
401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
"Please check your user name and/or password to be correct.",
403: "\nThere seems to be an issue with the access rights of the given user and the resource you queried.\n"
@@ -136,7 +138,7 @@ class APIEndpoint(ABC):
status_code = err.response.status_code
# get custom error message based on the returned status code
- error_message = error_code_to_message[status_code]
+ error_message = _error_code_to_message[status_code]
if status_code in [
HTTPStatus.METHOD_NOT_ALLOWED.numerator,
[incubator-streampipes] 13/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 2c1fe91f1dff9ce546c9ea550e4b77cdf37b9a93
Author: bossenti <bo...@posteo.de>
AuthorDate: Fri Nov 11 20:00:12 2022 +0100
tmp
---
streampipes-client-python/.pre-commit-config.yaml | 6 +++---
.../streampipes_client/{ => client}/endpoint/endpoint.py | 13 +++++++++++--
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/streampipes-client-python/.pre-commit-config.yaml b/streampipes-client-python/.pre-commit-config.yaml
index 154dde3fd..c876c23f3 100644
--- a/streampipes-client-python/.pre-commit-config.yaml
+++ b/streampipes-client-python/.pre-commit-config.yaml
@@ -40,11 +40,11 @@ repos:
types: [python]
entry: isort
- - id: flake8
- name: flake8
+ - id: ruff
+ name: ruff
language: python
types: [python]
- entry: flake8
+ entry: ruff
- id: mypy
name: mypy
diff --git a/streampipes-client-python/streampipes_client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
similarity index 89%
rename from streampipes-client-python/streampipes_client/endpoint/endpoint.py
rename to streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index 4bc54df19..9bf2f1984 100644
--- a/streampipes-client-python/streampipes_client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -14,6 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+
+"""
+General implementation for an endpoint.
+Provided classes and assets are aimed to be used for developing endpoints.
+An endpoint is provides all options to communicate with a central endpoint of the StreamPipes API in a handy way.
+"""
+
from http import HTTPStatus
import logging
from abc import ABC, abstractmethod
@@ -27,11 +34,13 @@ from streampipes_client.model.element import Element
logger = logging.getLogger(__name__)
+
+# define custom logging messages for some specific HTTP status
status_code_to_log_message = {
401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
"Please check your user name and/or password to be correct.",
- 403: "\nThere seems to be an issue with the accees rights of the given user and the ressource you queried.\n"
- "Apparently, this user is not allowd to query the resource.\n"
+ 403: "\nThere seems to be an issue with the access rights of the given user and the ressource you queried.\n"
+ "Apparently, this user is not allowed to query the resource.\n"
"Please check the user's permissions or contact your StreamPipes admin.",
**dict.fromkeys(
[404, 405],
[incubator-streampipes] 07/18: [#STREAMPIPES-622] Modify feature texts
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit f29670b870b40483a437364a356322d4a1ec5794
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Wed Nov 9 22:19:36 2022 +0100
[#STREAMPIPES-622] Modify feature texts
---
README.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index cbcd705ca..8ffe09149 100644
--- a/README.md
+++ b/README.md
@@ -79,17 +79,17 @@ Just write your own data processor and make it reusable as a pipeline element.
Besides that, StreamPipes includes features for production deployments:
* Assign resources such as pipelines, data streams and dashboards to assets for better organization
+* Monitoring & metrics of pipelines and adapters
* Built-in user and access rights management
-* User self-registration and email support
* Export and import resources
## User interface
-* Connect data from an OPC-UA server:
+* Connect data from an OPC-UA server following a three-step configuration process:

-* Create a pipeline to detect a trend:
+* Create a pipeline to detect a continuous decrease using a trend detection data processor and a ``Notification``sink:

@@ -116,7 +116,7 @@ For a more in-depth manual, read the [installation guide](https://streampipes.ap
## Documentation
-The full documentationis available [here](https://streampipes.apache.org/docs/index.html).
+The full documentation is available [here](https://streampipes.apache.org/docs/index.html).
Quick Links:
[incubator-streampipes] 17/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit ce6cde8346b2b9a7a2c259f4d714f20757e45307
Author: bossenti <bo...@posteo.de>
AuthorDate: Sat Nov 12 13:34:07 2022 +0100
tmp
---
.../streampipes_client/__init__.py | 8 ++
.../streampipes_client/client/__init__.py | 6 +
.../streampipes_client/client/client.py | 123 +++++++++++++++++++--
.../client/endpoint/data_lake_measure.py | 78 ++++++++++++-
.../streampipes_client/client/endpoint/endpoint.py | 40 ++++---
.../streampipes_client/model/container/__init__.py | 1 +
6 files changed, 225 insertions(+), 31 deletions(-)
diff --git a/streampipes-client-python/streampipes_client/__init__.py b/streampipes-client-python/streampipes_client/__init__.py
index 7bb805d49..285906ad1 100644
--- a/streampipes-client-python/streampipes_client/__init__.py
+++ b/streampipes-client-python/streampipes_client/__init__.py
@@ -18,3 +18,11 @@
"""
This library provides a handy Python-based client to interact with Apache StreamPipes.
"""
+
+from .client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
+
+__all = [
+ "StreamPipesApiKeyCredentials"
+ "StreamPipesClient",
+ "StreamPipesClientConfig",
+]
diff --git a/streampipes-client-python/streampipes_client/client/__init__.py b/streampipes-client-python/streampipes_client/client/__init__.py
index 47db35729..98d1d0e37 100644
--- a/streampipes-client-python/streampipes_client/client/__init__.py
+++ b/streampipes-client-python/streampipes_client/client/__init__.py
@@ -21,3 +21,9 @@ This module contains everything about the client itself, the central point of in
from .client import StreamPipesClient, StreamPipesClientConfig
from .credentials import StreamPipesApiKeyCredentials
+
+__all__ = [
+ "StreamPipesApiKeyCredentials",
+ "StreamPipesClient",
+ "StreamPipesClientConfig",
+]
diff --git a/streampipes-client-python/streampipes_client/client/client.py b/streampipes-client-python/streampipes_client/client/client.py
index 51323d947..9e7e2b471 100644
--- a/streampipes-client-python/streampipes_client/client/client.py
+++ b/streampipes-client-python/streampipes_client/client/client.py
@@ -14,6 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+
+"""
+Implementation of the StreamPipes client.
+The client is designed as the central point of interaction with the StreamPipes API and
+provides all functionalities to communicate with the API.
+"""
+
import logging
import sys
from dataclasses import dataclass
@@ -28,33 +35,102 @@ logger = logging.getLogger(__name__)
@dataclass
class StreamPipesClientConfig:
- """ """
+ """Configure the StreamPipes client in accordance to the actual StreamPipes instance to connect to.
+ An instance is provided to the `StreamPipesClient` to configure it properly.
+
+ Parameters
+ ----------
+ credential_provider: CredentialProvider
+ Provides the credentials to authenticate with the StreamPipes API.
+ host_address:
+ Host address of the StreamPipes instance to connect to.
+ Should be provided without the protocol/scheme, e.g. as `localhost` or `streampipes.xyz`.
+ https_disabled: Optional[bool]
+ Determines whether https is used to connect to StreamPipes.
+ port: Optional[int]
+ Specifies the port under which the StreamPipes API is available, e.g., 80.
+
+ Examples
+ --------
+ see `StreamPipesClient`
+ """
credential_provider: CredentialProvider
host_address: str
https_disabled: Optional[bool] = False
- port: Optional[int] = 80
+ port: Optional[int] = 80 # TODO: ask Dominik for general port, adapt doc strings as well
class StreamPipesClient:
+ """The client to connect to StreamPipes.
+ This is the central point of contact with StreamPipes and
+ provides all the functionalities to interact with it.
+
+ The client provides so-called "endpoints" each of which refers to
+ an endpoint of the StreamPipes API, e.g. `.dataLakeMeasureApi`.
+ An endpoint provides the actual methods to interact with StreamPipes API (see client.endpoint.Endpoint).
+
+ Parameters
+ ----------
+ client_config: StreamPipesClientConfig
+ Configures the client to connect properly to the StreamPipes instance.
+ logging_level: Optional[int]
+ Influences the log messages emitted by the `StreamPipesClient`.
+
+ Examples
+ --------
+
+ >>> from streampipes_client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
+
+ >>> client_config = StreamPipesClientConfig(
+ ... credential_provider=StreamPipesApiKeyCredentials(
+ ... username="test-user",
+ ... api_key="api-key"
+ ... ),
+ ... host_address="localhost",
+ ... https_disabled=True
+ ... )
+
+ >>> client = StreamPipesClient.create(client_config=client_config)
+
+ # The above way of instantiating a client instance is intended
+ # to be consistent with the StreamPipes Java client.
+ # If you prefer a more pythonic way, you can simply write:
+ >>> client = StreamPipesClient(client_config=client_config)
+
+ >>> data_lake_measures = client.dataLakeMeasureApi.all()
+ """
def __init__(
self,
- *,
client_config: StreamPipesClientConfig,
- logging_level: int = logging.INFO,
+ logging_level: Optional[int] = logging.INFO,
):
self.client_config = client_config
+ # set up a requests session
+ # this allows to centrally determine the behavior of all requests made
self.request_session = requests.Session()
self.request_session.headers.update(self.http_headers)
self._set_up_logging(logging_level=logging_level)
- # endpoints
+ # provide all available endpoints here
+ # name of the endpoint needs to be consistent with the Java client
self.dataLakeMeasureApi = DataLakeMeasureEndpoint(parent_client=self)
@staticmethod
- def _set_up_logging(*, logging_level: int) -> None:
+ def _set_up_logging(logging_level: int) -> None:
+ """Configures the logging behavior of the `StreamPipesClient`.
+
+ Parameters
+ ----------
+ logging_level: Optional[int]
+ Influences the log messages emitted by the `StreamPipesClient`.
+
+ Returns
+ -------
+ None
+ """
logging.basicConfig(
level=logging_level,
stream=sys.stdout,
@@ -66,18 +142,49 @@ class StreamPipesClient:
@classmethod
def create(
cls,
- *,
client_config: StreamPipesClientConfig,
logging_level: int = logging.INFO,
):
+ """Returns an instance of the `StreamPipesPythonClient`.
+ Provides consistency to the Java client.
+
+ Parameters
+ ----------
+ client_config: StreamPipesClientConfig
+ Configures the client to connect properly to the StreamPipes instance.
+ logging_level: Optional[int]
+ Influences the log messages emitted by the `StreamPipesClient`.
+
+ Returns
+ -------
+ StreamPipesClient
+ """
return cls(client_config=client_config, logging_level=logging_level)
@property
def http_headers(self) -> Dict[str, str]:
- return self.client_config.credential_provider.make_headers({"Application": "application/json"})
+ """Returns the HTTP headers required for all requests.
+ The HTTP headers are composed of the authentication headers provided by the credential
+ provider and additional required headers (currently this is only the application header).
+
+ Returns
+ -------
+ Dictionary with header information as string key-value pairs
+ """
+
+ # create HTTP headers from credential provider and add additional headers needed
+ return self.client_config.credential_provider.make_headers(
+ {"Application": "application/json"},
+ )
@property
def base_api_path(self) -> str:
+ """Constructs the basic API URL from the given `client_config`.
+
+ Returns
+ -------
+ str of the basic API URL
+ """
return (
f"{'http://' if self.client_config.https_disabled else 'https://'}"
f"{self.client_config.host_address}:"
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
index 76f5a1c2a..a46338165 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/data_lake_measure.py
@@ -1,12 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Specific implementation of the StreamPipes API's data lake measure endpoints.
+This endpoint allows to consume data stored in StreamPipes' data lake
+"""
from client.endpoint.endpoint import APIEndpoint
from streampipes_client.model.container.data_lake_measures import DataLakeMeasures
+__all__ = [
+ "DataLakeMeasureEndpoint",
+]
+
class DataLakeMeasureEndpoint(APIEndpoint):
- @classmethod
- def _container_cls(cls):
+ """Implementation of the DataLakeMeasure endpoint.
+ This endpoint provides an interfact to all data stored in the StreamPipes data lake.
+
+ Consequently, it allows uerying metadata about available data sets (see `all()` method).
+ The metadata is returned as an instance of `model.container.DataLakeMeasures`.
+
+ In addition, the endpoint provides direct access to the data stored in the data laka by querying a
+ specific data lake measure using the `get()` method.
+
+ Parameters
+ ----------
+ parent_client: `client.StreamPipesClient`
+ The instance of `StreamPipesClient` the endpoint is attached to.
+
+ Examples
+ --------
+
+ >>> from streampipes_client import StreamPipesClient, StreamPipesClientConfig, StreamPipesApiKeyCredentials
+
+ >>> client_config = StreamPipesClientConfig(
+ ... credential_provider=StreamPipesApiKeyCredentials(username="test-user", api_key="api-key"),
+ ... host_address="localhost",
+ ... port=8082,
+ ... https_disabled=True
+ ... )
+
+ >>> client = StreamPipesClient.create(client_config=client_config)
+
+ >>> data_lake_measures = client.dataLakeMeasureApi.all()
+
+ >>> len(data_lake_measures)
+ 5
+ """
+
+ @property
+ def _container_cls(self):
+ """Defines the model container class the endpoint refers to.
+
+
+ Returns
+ -------
+ `model.container.DataLakeMeasures`
+ """
return DataLakeMeasures
@property
def _relative_api_path(self):
+ """Defines the relative api path to the DataLakeMeasurement endpoint.
+ Each path within the URL is defined as an own string.
+
+ Returns
+ -------
+ A tuple of strings of which every represents a path value of the endpoint's API URL.
+ """
+
return "api", "v4", "datalake", "measurements"
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index b4c84b37e..7f349b386 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -40,10 +40,10 @@ logger = logging.getLogger(__name__)
# define custom logging messages for some specific HTTP status
_error_code_to_message = {
401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
- "Please check your user name and/or password to be correct.",
+ "Please check your user name and/or password to be correct.",
403: "\nThere seems to be an issue with the access rights of the given user and the resource you queried.\n"
- "Apparently, this user is not allowed to query the resource.\n"
- "Please check the user's permissions or contact your StreamPipes admin.",
+ "Apparently, this user is not allowed to query the resource.\n"
+ "Please check the user's permissions or contact your StreamPipes admin.",
**dict.fromkeys(
[404, 405],
"\nOops, there seems to be an issue with the Python Client calling the API inappropriately.\n"
@@ -74,35 +74,35 @@ class APIEndpoint(ABC):
@property
@abstractmethod
- def _relative_api_path(self) -> Tuple[str]:
- """Defines the relative api path with regard to the StreamPipes API URL.
- Each path within the URL is defined as an own string.
+ def _container_cls(cls) -> Type[ModelContainer]:
+ """Defines the model container class the endpoint refers to.
+ This model container class corresponds to the Python data model,
+ which handles multiple resources returned from the endpoint.
Returns
-------
- A tuple of strings of which every represents a path value of the endpoint's API URL.
-
+ The corresponding container class from the data model,
+ needs to a subclass of `model.ModelContainer`.
"""
raise NotImplementedError
- @classmethod
+ @property
@abstractmethod
- def _container_cls(cls) -> Type[ModelContainer]:
- """Defines the model container class the endpoint refers to.
- This model container class corresponds to thePython data model,
- which handles multiple resources returned from the endpoint.
+ def _relative_api_path(self) -> Tuple[str]:
+ """Defines the relative api path with regard to the StreamPipes API URL.
+ Each path within the URL is defined as an own string.
Returns
-------
- The corresponding container class from the data model,
- needs to a subclass of `model.ModelContainer`.
+ A tuple of strings of which every represents a path value of the endpoint's API URL.
+
"""
raise NotImplementedError
@staticmethod
def _make_request(
- request_method: Callable[..., Response],
- url: str,
+ request_method: Callable[..., Response],
+ url: str,
) -> Response:
"""Helper method to send requests to the StreamPipes API endpoint.
Should be used from methods of this class that interacts with the API, e.g. `all()` and `get()`.
@@ -192,7 +192,5 @@ class APIEndpoint(ABC):
-------
The specified resource as an instance of the corresponding model class (`model.Element`).
"""
-
- # equals to download
- # needs further considerations
- pass
+ raise NotImplementedError("We're sorry! This functionality is not yet part of the StreamPipes Python client."
+ "Stay tuned, we will add this shortly.")
diff --git a/streampipes-client-python/streampipes_client/model/container/__init__.py b/streampipes-client-python/streampipes_client/model/container/__init__.py
index 9d8d2b197..aa5f2a267 100644
--- a/streampipes-client-python/streampipes_client/model/container/__init__.py
+++ b/streampipes-client-python/streampipes_client/model/container/__init__.py
@@ -16,3 +16,4 @@
#
from .model_container import ModelContainer
+from .data_lake_measures import DataLakeMeasures
[incubator-streampipes] 11/18: test
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit fb705eb4afb148d81e3010fd84b0f90530e5367b
Author: bossenti <bo...@posteo.de>
AuthorDate: Thu Nov 10 21:29:35 2022 +0100
test
---
README.md | 2 +-
streampipes-client-python/README.md | 14 +++++++++-----
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/README.md b/README.md
index 90cacb1ed..1805e5f55 100644
--- a/README.md
+++ b/README.md
@@ -70,7 +70,7 @@ It comes with a rich graphical user interface targeted at non-technical users an
StreamPipes is highly extensible and includes a Java SDK to create new
-pipeline elements and adapters. Python support is available in an early development stage - stay tuned!
+pipeline elements and adapters. [Python support](./streampipes-client-python/README.md) is available in an early development stage - stay tuned!
Pipeline elements are standalone microservices that can run anywhere -
centrally on your server or close at the edge.
You want to employ your own machine learning model on live data?
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index 6cf8a67be..4fbb650c5 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -19,12 +19,15 @@
# StreamPipes client for Python
Apache StreamPipes meets Python! We are working highly motivated on a Python-based client to interact with StreamPipes.
-Thus, we would like to unite the power of StreamPipes to easily connect to and read from several data sources, especially in the IoT domain,
-and the amazing university of data analytics libraries in Python.
+In this way, we would like to unite the power of StreamPipes to easily connect to and read different data sources, especially in the IoT domain,
+and the amazing universe of data analytics libraries in Python.
-<br>
-
-
+<h1 align="center">
+ <br>
+ <img src="./docs/img/streampipes-python.png"
+ alt="StreamPipes Logo with Python" title="Apache StreamPipes Logo with Python" width="75%"/>
+ <br>
+</h1>
<br>
@@ -46,6 +49,7 @@ and the amazing university of data analytics libraries in Python.
<br>
Create a virtual Python environment with a tool of your choice, here are some examples.
As a next step, install all required dependencies for the development, e.g. with pip:
+
```
pip install .[dev] # or alternatively: pip install .[docs]
```
[incubator-streampipes] 08/18: [STREAMPIPES-622] Modify introduction text
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit a418be586e64abba2d47a59a294e83c4c3aadc09
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Wed Nov 9 22:21:54 2022 +0100
[STREAMPIPES-622] Modify introduction text
---
README.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index 8ffe09149..06e681529 100644
--- a/README.md
+++ b/README.md
@@ -63,14 +63,14 @@ StreamPipes is an end-to-end toolbox for the industrial IoT.
It comes with a rich graphical user interface targeted at non-technical users and provides the following features:
-* Quickly connect >20 industrial data sources and protocols such as OPC-UA, PLCs, MQTT, REST, Pulsar, Kafka and others
-* Create data harmonization and analytics pipelines using > 100 algorithms and many data sinks to forward data to third-party systems
+* Quickly connect >20 industrial protocols such as OPC-UA, PLCs, MQTT, REST, Pulsar, Kafka and others.
+* Create data harmonization and analytics pipelines using > 100 algorithms and data sinks to forward data to third-party systems.
* Use the data explorer to visually explore historical data with many widgets tailored for time-series data.
* A live dashboard to display real-time data from data sources and pipelines, e.g., for shopfloor monitoring.
-StreamPipes is highly extensible and includes a Java and Python (currently in development phase) SDK to create new
-pipeline elements and adapters.
+StreamPipes is highly extensible and includes a Java SDK to create new
+pipeline elements and adapters. Python support is available in an early development stage - stay tuned!
Pipeline elements are standalone microservices that can run anywhere -
centrally on your server or close at the edge.
You want to employ your own machine learning model on live data?
[incubator-streampipes] 10/18: [hotfix] Update contribution section in README
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit cd0a54fb247b911008db219dcbf6e2c7a70e5a63
Author: Philipp Zehnder <te...@users.noreply.github.com>
AuthorDate: Thu Nov 10 09:25:58 2022 +0100
[hotfix] Update contribution section in README
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index a593daeb0..90cacb1ed 100644
--- a/README.md
+++ b/README.md
@@ -191,12 +191,12 @@ Or directly subscribe to [users-subscribe@streampipes.apache.org](mailto:users-s
## Contribute
-We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll
+We welcome all kinds of contributions to StreamPipes. If you are interested in contributing, let us know! You'll
get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.
Here are some first steps in case you want to contribute:
* Subscribe to our dev mailing list [dev-subscribe@streampipes.apache.org](dev-subscribe@streampipes.apache.org)
-* Send an email, tell us about your interests and which parts of Streampipes you'd like to contribute (e.g., core or UI)!
+* Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or UI)!
* Ask for a mentor who helps you understanding the code base and guides you through the first setup steps
* Find an issue in our [Jira](https://issues.apache.org/jira/projects/STREAMPIPES) which is tagged with a _newbie_ tag
* Have a look at our developer wiki at [https://cwiki.apache.org/confluence/display/STREAMPIPES](https://cwiki.apache.org/confluence/display/STREAMPIPES) to learn more about StreamPipes development.
[incubator-streampipes] 15/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit e6151d27728fb78e2d42da42b7609cd5401d9623
Author: bossenti <bo...@posteo.de>
AuthorDate: Sat Nov 12 08:22:43 2022 +0100
tmp
---
.../streampipes_client/client/endpoint/endpoint.py | 86 +++++++++++++++++-----
1 file changed, 68 insertions(+), 18 deletions(-)
diff --git a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
index 8547c69bf..a113501b7 100644
--- a/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
+++ b/streampipes-client-python/streampipes_client/client/endpoint/endpoint.py
@@ -24,7 +24,7 @@ An endpoint is provides all options to communicate with a central endpoint of th
import logging
from abc import ABC, abstractmethod
from http import HTTPStatus
-from typing import Callable, Tuple, Type
+from typing import Callable, Tuple, Type, Union
from requests import Response
from requests.exceptions import BaseHTTPError, HTTPError
@@ -35,9 +35,8 @@ __all__ = ["APIEndpoint"]
logger = logging.getLogger(__name__)
-
# define custom logging messages for some specific HTTP status
-status_code_to_log_message = {
+error_code_to_message = {
401: "\nThe StreamPipes Backend returned an unauthorized error.\n"
"Please check your user name and/or password to be correct.",
403: "\nThere seems to be an issue with the access rights of the given user and the resource you queried.\n"
@@ -74,13 +73,12 @@ class APIEndpoint(ABC):
@property
@abstractmethod
def _relative_api_path(self) -> Tuple[str]:
- """
- Defines the relative api path with regard to the StreamPipes API URL.
+ """Defines the relative api path with regard to the StreamPipes API URL.
Each path within the URL is defined as an own string.
Returns
-------
- A tuple of strings of which every represents a path value of the endpoint's API URL.
+ A tuple of strings of which every represents a path value of the endpoint's API URL.
"""
raise NotImplementedError
@@ -88,39 +86,66 @@ class APIEndpoint(ABC):
@classmethod
@abstractmethod
def _container_cls(cls) -> Type[ModelContainer]:
- """
+ """Defines the model container class the endpoint refers to.
+ This model container class corresponds to thePython data model,
+ which handles multiple resources returned from the endpoint.
Returns
-------
- A class which inherits from `model.ModelContainer`
+ The corresponding container class from the data model,
+ needs to a subclass of `model.ModelContainer`.
"""
raise NotImplementedError
+ @staticmethod
def _make_request(
- self,
- *,
request_method: Callable[..., Response],
url: str,
) -> Response:
+ """Helper method to send requests to the StreamPipes API endpoint.
+ Should be used from methods of this class that interacts with the API, e.g. `all()` and `get()`.
+
+ Parameters
+ ----------
+ request_method: Callable[..., Response]
+ The HTTP method with which to submit the request.
+ Must be one of HTTP methods provided by the `requests` library, e.g. `requests.get`.
+ url: str
+ The full URL to which the request should be applied.
+
+ Returns
+ -------
+ An HTTP response, which is of type `requests.Response` and
+ contains both the actual API response and some metadata.
+ Returned only if the request was successful,
+ otherwise it raises an exception (see `Raises`).
+
+ Raises
+ ------
+ requests.exceptions.HTTPError
+ If the HTTP status code of the error is between `400` and `600`.
+ """
response = request_method(url=url)
+ # check if the API request was successful
try:
response.raise_for_status()
except HTTPError as err:
status_code = err.response.status_code
- log_message = status_code_to_log_message[err.response.status_code]
+ # get custom error message based on the returned status code
+ error_message = error_code_to_message[status_code]
if status_code in [
HTTPStatus.METHOD_NOT_ALLOWED.numerator,
HTTPStatus.NOT_FOUND.numerator,
]:
- log_message += f"url: {err.response.url}\nstatus code: {err.response.status_code}"
+ error_message += f"url: {err.response.url}\nstatus code: {status_code}"
- logger.debug(err.response.text)
- raise HTTPError(log_message) from err
+ logger.debug(f"HTTP error response: {err.response.text}")
+ raise HTTPError(error_message) from err
else:
logger.debug("Successfully retrieved resources from %s.", url)
@@ -128,18 +153,43 @@ class APIEndpoint(ABC):
return response
- def create_api_path(self) -> str:
- return f"{self._parent_client.base_api_path}{'/'.join(api_path for api_path in self._relative_api_path)}"
+ def build_url(self) -> str:
+ """Creates the URL of the API path for the endpoint.
+
+ Returns
+ -------
+ The URL of the Endpoint
+ """
+ return f"{self._parent_client.base_api_path}" f"{'/'.join(api_path for api_path in self._relative_api_path)}"
def all(self) -> ModelContainer:
+ """Get all resources of this endpoint provided by the StreamPipes API.
+ Results are provided as an instance of a `model.ModelContainer` that
+ allows to handle the returned resources in a comfortable and pythonic way.
+
+ Returns
+ -------
+ A model container instance (`model.ModelContainer`) bundling the resources returned.
+ """
response = self._make_request(
request_method=self._parent_client.request_session.get,
- url=self.create_api_path(),
+ url=self.build_url(),
)
return self._container_cls().from_json(json_string=response.text)
- def get(self, *, identifier: str) -> Element:
+ def get(self, identifier: str) -> Element:
+ """Queries the specified resource from the API endpoint.
+
+ Parameters
+ ----------
+ identifier: str
+ The identifier of the resource to be queried.
+
+ Returns
+ -------
+ The specified resource as an instance of the corresponding model class (`model.Element`).
+ """
# equals to download
# needs further considerations
[incubator-streampipes] 02/18: tmp
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 5ccd988d64634ee286c53530a70939e6dcd2ff78
Author: bossenti <bo...@posteo.de>
AuthorDate: Tue Nov 1 15:50:31 2022 +0100
tmp
---
.../streampipesclient/__init__.py | 18 ++++++++
.../streampipesclient/client/__init__.py | 20 +++++++++
.../streampipesclient/client/client.py | 30 +++++++++++++
.../streampipesclient/client/config.py | 32 ++++++++++++++
.../streampipesclient/client/credentials.py | 50 ++++++++++++++++++++++
5 files changed, 150 insertions(+)
diff --git a/streampipes-client-python/streampipesclient/__init__.py b/streampipes-client-python/streampipesclient/__init__.py
new file mode 100644
index 000000000..43a8ab2c8
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from .client import StreamPipesClient
diff --git a/streampipes-client-python/streampipesclient/client/__init__.py b/streampipes-client-python/streampipesclient/client/__init__.py
new file mode 100644
index 000000000..56ed17e8a
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/client/__init__.py
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from .client import StreamPipesClient
+from .config import StreamPipesClientConfig
+from .credentials import StreamPipesApiKeyCredentials
diff --git a/streampipes-client-python/streampipesclient/client/client.py b/streampipes-client-python/streampipesclient/client/client.py
new file mode 100644
index 000000000..f2855bbeb
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/client/client.py
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+class StreamPipesClient:
+
+ def __init__(self):
+
+ pass
+
+ @property
+ def http_headers(self):
+ pass
+
+ @property
+ def base_api_path(self):
+ pass
\ No newline at end of file
diff --git a/streampipes-client-python/streampipesclient/client/config.py b/streampipes-client-python/streampipesclient/client/config.py
new file mode 100644
index 000000000..b1f62fe24
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/client/config.py
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from dataclasses import dataclass
+from typing import Optional, Type
+
+from streampipesclient.client.credentials import CredentialProvider
+
+
+@dataclass
+class StreamPipesClientConfig:
+ """
+
+ """
+
+ credential_provider: Type[CredentialProvider]
+ host_address: str
+ https_disabled: Optional[bool] = False
+ port: Optional[int] = 80
diff --git a/streampipes-client-python/streampipesclient/client/credentials.py b/streampipes-client-python/streampipesclient/client/credentials.py
new file mode 100644
index 000000000..fe2cc2b89
--- /dev/null
+++ b/streampipes-client-python/streampipesclient/client/credentials.py
@@ -0,0 +1,50 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from abc import ABC, abstractmethod
+from typing import Dict, Optional
+
+
+class CredentialProvider(ABC):
+
+ @abstractmethod
+ def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
+ raise NotImplementedError
+
+
+class StreamPipesApiKeyCredentials(CredentialProvider):
+
+ def __init__(self,
+ *,
+ username: str,
+ api_key: str,
+ ):
+ self.username = username
+ self.api_key = api_key
+
+ def make_headers(self, http_headers: Optional[Dict[str, str]] = None) -> Dict[str, str]:
+
+ if http_headers is None:
+ http_headers = {}
+
+ http_headers.update(
+ {
+ "X-API-User": self.username,
+ "X-API-Key": self.api_key
+ }
+ )
+
+ return http_headers
[incubator-streampipes] 05/18: [STREAMPIPES-622] Modify README intro text, add animated screenshots
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit f1c993bbebd80c75d40c560f43a63e4972a02f92
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Tue Nov 8 23:55:09 2022 +0100
[STREAMPIPES-622] Modify README intro text, add animated screenshots
---
README.md | 64 ++++++++++++++++++++++-------------
images/streampipes-connect.gif | Bin 0 -> 949971 bytes
images/streampipes-data-explorer.gif | Bin 0 -> 543486 bytes
images/streampipes-pipelines.gif | Bin 0 -> 1296806 bytes
4 files changed, 41 insertions(+), 23 deletions(-)
diff --git a/README.md b/README.md
index b054d0ad7..bb53b65e1 100644
--- a/README.md
+++ b/README.md
@@ -23,6 +23,8 @@
[]()
[](https://twitter.com/StreamPipes)
[](https://dashboard.cypress.io/projects/q1jdu2/runs)
+[](https://github.com/apache/incubator-streampipes/graphs/contributors)
+
<h1 align="center">
<br>
@@ -33,9 +35,6 @@
<h3 align="center">Self-Service Data Analytics for the (Industrial) IoT</h3>
<h4 align="center">StreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect
, analyze and explore IoT data streams. </h4>
-<p align="center">
- <img src="https://streampipes.apache.org/img/screenshots/streampipes-overview.png" alt="StreamPipes Pipeline
- Editor"/>
</p>
***
@@ -43,8 +42,9 @@
## Table of contents
* [About Apache StreamPipes](#about-apache-streampipes)
- * [Use Cases](#use-cases)
+ * [User interface](#userinterface)
* [Installation](#installation)
+ * [Documentation](#documentation)
* [Building StreamPipes](#building-streampipes)
* [Pipeline Elements](#pipeline-elements)
* [Extending StreamPipes](#extending-streampipes)
@@ -58,30 +58,42 @@
## About Apache StreamPipes
-Apache StreamPipes (incubating) enables flexible modeling of stream processing pipelines by providing a graphical
-modeling editor on top of existing stream processing frameworks.
+Apache StreamPipes (incubating) makes industrial data analytics easy!
-It empowers non-technical users to quickly define and execute processing pipelines based on an easily extensible
-toolbox of data sources, data processors and data sinks. StreamPipes has an exchangeable runtime execution layer and
-executes pipelines using one of the provided wrappers, e.g., standalone or distributed in Apache Flink.
+StreamPipes is an end-to-end toolbox for the industrial IoT.
+It comes with a rich graphical user interface targeted at non-technical users and provides the following features:
-Pipeline elements in StreamPipes can be installed at runtime - the built-in SDK allows to easily implement new
-pipeline elements according to your needs. Pipeline elements are standalone microservices that can run anywhere -
-centrally on your server, in a large-scale cluster or close at the edge.
+* Quickly connect >20 industrial data sources and protocols such as OPC-UA, PLCs, MQTT, REST, Kafka and others
+* Create data harmonization and analytics pipelines using > 100 algorithms and many data sinks to forward data to third-party systems
+* Use the data explorer to visually explore historical data with many widgets tailored for time-series data.
+* A live dashboard to display real-time data from data sources and pipelines, e.g., for shopfloor monitoring.
-## Use Cases
+StreamPipes is highly extensible and includes a Java and Python (currently in development phase) SDK to create new
+pipeline elements and adapters.
+Pipeline elements are standalone microservices that can run anywhere -
+centrally on your server or close at the edge.
+You want to employ your own machine learning model on live data?
+Just write your own data processor and make it reusable as a pipeline element.
-StreamPipes allows you to connect IoT data sources using the SDK or the built-in graphical tool **StreamPipes
-Connect**.
+Besides that, StreamPipes includes features for production deployments:
+* Assign resources such as pipelines, data streams and dashboards to assets for better organization
+* Built-in user and access rights management
+* User self-registration and email support
+* Export and import resources
-The extensible toolbox of data processors and sinks supports use cases such as
-* Continuously **store** IoT data streams to third party systems (e.g., databases)
-* **Filter** measurements on streams (e.g., based on thresholds or value ranges)
-* **Harmonize** data by using data processors for transformations (e.g., by converting measurement units and data types
- or by aggregating measurements)
-* **Detect situations** that should be avoided (e.g., patterns based on time windows)
-* Wrap **Machine Learning models** into data processors to perform classifications or predictions on sensor and image data
-* **Visualize** real-time data from sensors and machines using the built-in Live Dashboard
+## User interface
+
+* Connect data from an OPC-UA server:
+
+
+
+* Create a pipeline to detect a trend:
+
+
+
+* Visually analyze data using the data explorer:
+
+
## Installation
@@ -98,6 +110,12 @@ Please follow the instructions provided in the corresponding `README.md` to get
For a more in-depth manual, read the [installation guide](https://streampipes.apache.org/docs/docs/try-installation.html).
+> TL;DR: Download the latest release, switch to the ``installer/compose`` directory and run ``docker-compose up -d``.
+
+## Documentation
+
+The full documentation, including an overview of pipeline elements, a developer guide and several tutorials is available at [https://streampipes.apache.org/docs/docs/user-guide-introduction.html](https://streampipes.apache.org/docs/docs/user-guide-introduction.html)
+
## Building StreamPipes
To properly build the StreamPipes core, the following tools should be installed:
diff --git a/images/streampipes-connect.gif b/images/streampipes-connect.gif
new file mode 100644
index 000000000..33fd62a65
Binary files /dev/null and b/images/streampipes-connect.gif differ
diff --git a/images/streampipes-data-explorer.gif b/images/streampipes-data-explorer.gif
new file mode 100644
index 000000000..7f1ba0755
Binary files /dev/null and b/images/streampipes-data-explorer.gif differ
diff --git a/images/streampipes-pipelines.gif b/images/streampipes-pipelines.gif
new file mode 100644
index 000000000..abc93150a
Binary files /dev/null and b/images/streampipes-pipelines.gif differ
[incubator-streampipes] 01/18: [STREAMPIPES-606] create module for python client
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit e8a771bf19b718121b6d6ab5f7cb653ecfd987be
Author: bossenti <bo...@posteo.de>
AuthorDate: Thu Oct 27 21:48:46 2022 +0200
[STREAMPIPES-606] create module for python client
---
streampipes-client-python/README.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index 8641b5f54..0884a3039 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -14,4 +14,5 @@
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
--->
\ No newline at end of file
+-->
+... upcoming Python client for StreamPipes, stay tuned!
\ No newline at end of file
[incubator-streampipes] 04/18: test
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch feature/STREAMPIPES-607
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes.git
commit 554f996330af2edaf015ecd3513a303a8523a2fb
Author: bossenti <bo...@posteo.de>
AuthorDate: Thu Nov 10 21:18:51 2022 +0100
test
---
streampipes-client-python/.pre-commit-config.yaml | 2 +-
streampipes-client-python/README.md | 46 ++++++++++++++++++++-
.../docs/img/streampipes-python.png | Bin 0 -> 192272 bytes
3 files changed, 46 insertions(+), 2 deletions(-)
diff --git a/streampipes-client-python/.pre-commit-config.yaml b/streampipes-client-python/.pre-commit-config.yaml
index 1eccb226e..154dde3fd 100644
--- a/streampipes-client-python/.pre-commit-config.yaml
+++ b/streampipes-client-python/.pre-commit-config.yaml
@@ -56,4 +56,4 @@ repos:
name: pyupgrade
language: python
types: [python]
- entry: pyupgrade --py38-plus
\ No newline at end of file
+ entry: pyupgrade --py38-plus --keep-runtime-typing
\ No newline at end of file
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index 0884a3039..6cf8a67be 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -15,4 +15,48 @@
~ limitations under the License.
~
-->
-... upcoming Python client for StreamPipes, stay tuned!
\ No newline at end of file
+
+# StreamPipes client for Python
+
+Apache StreamPipes meets Python! We are working highly motivated on a Python-based client to interact with StreamPipes.
+Thus, we would like to unite the power of StreamPipes to easily connect to and read from several data sources, especially in the IoT domain,
+and the amazing university of data analytics libraries in Python.
+
+<br>
+
+
+
+<br>
+
+**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
+<br>
+<br>
+**The current version of this Python client is still in alpha phase at best**
+<br>
+**This means that it is still heavily under development, which may result in frequent and extensive API changes, unstable behavior, etc.**
+<br>
+**Please consider it only as paa sneak preview.**
+<br>
+<br>
+**:exclamation::exclamation::exclamation:IMPORTANT:exclamation::exclamation::exclamation:**
+
+### Get ready for development
+
+1) Set up your Python environment
+<br>
+Create a virtual Python environment with a tool of your choice, here are some examples.
+As a next step, install all required dependencies for the development, e.g. with pip:
+```
+pip install .[dev] # or alternatively: pip install .[docs]
+```
+<br>
+2) Install pre-commit hook
+<br>
+The pre-commit hook is run before every commit and takes care about code style,
+linting, type hints, import sorting, etc.
+Ensure to have the recent version of the pre-commit installed otherwise the CI build might fail:
+
+```
+pre-commit install
+```
+The definition of the pre-commit hook can found in [.pre-commit-config.yaml](.pre-commit-config.yaml).
\ No newline at end of file
diff --git a/streampipes-client-python/docs/img/streampipes-python.png b/streampipes-client-python/docs/img/streampipes-python.png
new file mode 100644
index 000000000..8cdebdb14
Binary files /dev/null and b/streampipes-client-python/docs/img/streampipes-python.png differ