You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by bo...@apache.org on 2022/12/04 20:44:15 UTC
[streampipes] branch add-example-files-python-client created (now f3181c3f6)
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a change to branch add-example-files-python-client
in repository https://gitbox.apache.org/repos/asf/streampipes.git
at f3181c3f6 add examples for Python client
This branch includes the following new commits:
new e3cac9dd6 add describe() method to python client
new f3181c3f6 add examples for Python client
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[streampipes] 02/02: add examples for Python client
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch add-example-files-python-client
in repository https://gitbox.apache.org/repos/asf/streampipes.git
commit f3181c3f6a07f3ba2985010b93e0dabe35dbfd80
Author: bossenti <bo...@posteo.de>
AuthorDate: Sun Dec 4 21:43:57 2022 +0100
add examples for Python client
---
streampipes-client-python/README.md | 34 +-
...introduction-to-streampipes-python-client.ipynb | 213 +++++++++++
...cting-data-from-the-streampipes-data-lake.ipynb | 391 +++++++++++++++++++++
.../docs/getting-started/installation.md | 2 +-
.../docs/getting-started/quickstart.md | 19 +-
streampipes-client-python/mkdocs.yml | 4 +
streampipes-client-python/setup.py | 1 +
7 files changed, 623 insertions(+), 41 deletions(-)
diff --git a/streampipes-client-python/README.md b/streampipes-client-python/README.md
index 9ed564210..836603dc6 100644
--- a/streampipes-client-python/README.md
+++ b/streampipes-client-python/README.md
@@ -53,7 +53,6 @@ and the amazing universe of data analytics libraries in Python. </p>
## ⚡️ Quickstart
As a quick example, we demonstrate how to set up and configure a StreamPipes client.
-In addition, we will get the available data lake measures out of StreamPipes.
```python
>>> from streampipes_client.client import StreamPipesClient
@@ -71,32 +70,13 @@ In addition, we will get the available data lake measures out of StreamPipes.
...)
>>> client = StreamPipesClient(client_config=config)
+>>> client.describe()
-# get all available datat lake measures
->>> measures = client.dataLakeMeasureApi.all()
-
-# get amount of retrieved measures
->>> len(measures)
-1
-
-# inspect the data lake measures as pandas dataframe
->>> measures.to_pandas()
- measure_name timestamp_field ... pipeline_is_running num_event_properties
-0 test s0::timestamp ... False 2
-[1 rows x 6 columns]
-```
-<br>
-Alternatively, you can provide your credentials via environment variables.
-Simply define your credential provider as follows:
-
-```python
->>> from streampipes_client.client.credential_provider import StreamPipesApiKeyCredentials
-
-StreamPipesApiKeyCredentials.from_env(username_env="USER", api_key_env="API-KEY")
+Hi there!
+You are connected to a StreamPipes instance running at http://localhost:80.
+The following StreamPipes resources are available with this client:
+6x DataStreams
+1x DataLakeMeasures
```
-<br>
-
-`username` is always the username that is used to log in into StreamPipes. <br>
-The `api_key` can be generated within the UI as demonstrated below:
-
\ No newline at end of file
+For more information about how to use the StreamPipes client visit our [introduction example]().
\ No newline at end of file
diff --git a/streampipes-client-python/docs/examples/1-introduction-to-streampipes-python-client.ipynb b/streampipes-client-python/docs/examples/1-introduction-to-streampipes-python-client.ipynb
new file mode 100644
index 000000000..4d215a983
--- /dev/null
+++ b/streampipes-client-python/docs/examples/1-introduction-to-streampipes-python-client.ipynb
@@ -0,0 +1,213 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Introduction to StreamPipes Python Client\n",
+ "\n",
+ "<br>\n",
+ "\n",
+ "### Why there is an extra Python client for StreamPipes\n",
+ "[Apache StreamPipes](https://streampipes.apache.org/) aims to enable non-technical users to connect and analyze IoT data streams.\n",
+ "To this end, it provides an easy-to-use and convenient user interface that allows one to connect to an IoT data source and create some visual\n",
+ "graphs within a few minutes. <br>\n",
+ "Although this is the main use case of Apache StreamPipes, it can also provide great value for people who are eager to work on data analysis or data science with IoT data, but don't we do get in touch with all the hassle associated with extracting data from devices in a suitable format.\n",
+ "In this scenario, StreamPipes helps you connect to your data source and extract the data for you.\n",
+ "You then can make the data available outside StreamPipes by writing it into an external source, such as a database, Kafka, etc.\n",
+ "While this requires another component, you can also extract your data directly from StreamPipes programmatically using the StreamPipes API.\n",
+ "For convenience, we also provide you with a StreamPipes client both available for Java and Python.\n",
+ "Specifically with the Python client, we want to address the amazing data analytics and data science community in Python and benefit from the great universe of Python libraries out there.\n",
+ "\n",
+ "<br>\n",
+ "\n",
+ "### How to install the Python client\n",
+ "Up to this point, we do not provide a release of the Python client in any of the package indexes known for Python.\n",
+ "This will probably start with StreamPipes `1.0.0` when we officially launch the Python client.\n",
+ "Until then, you can just install the currently available development version of the client directly from GitHub.\n",
+ "Simply use the following `pip` command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "%pip install git+https://github.com/apache/streampipes.git#subdirectory=streampipes-client-python"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### How to configure the Python client\n",
+ "In order to access the resources available in StreamPipes, one must be able to authenticate against the backend.\n",
+ "For this purpose, the client sofar only supports the authentication via an API token that can be generated via the StreamPipes UI, as you see below.\n",
+ "\n",
+ "\n",
+ "\n",
+ "<br>\n",
+ "\n",
+ "Having generated the API token, one can directly start initializing a client instance as follows:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "outputs": [],
+ "source": [
+ "from streampipes_client.client import StreamPipesClient\n",
+ "from streampipes_client.client.client_config import StreamPipesClientConfig\n",
+ "from streampipes_client.client.credential_provider import StreamPipesApiKeyCredentials"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "config = StreamPipesClientConfig(\n",
+ " credential_provider=StreamPipesApiKeyCredentials(\n",
+ " username=\"test@streampipes.apache.org\",\n",
+ " api_key=\"DEMO-KEY\",\n",
+ " ),\n",
+ " host_address=\"localhost\",\n",
+ " https_disabled=True,\n",
+ " port=80\n",
+ ")"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Please be aware that connecting to StreamPipes via a `https` connection is currently not supported by the Python client.\n",
+ "\n",
+ "Providing secrets like the `api_key` as plaintext in the source code is an anti-pattern.\n",
+ "This is why the StreamPipes client also supports passing the required secrets as environment variables.\n",
+ "To do so, you must initialize the credential provider like the following:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "StreamPipesApiKeyCredentials.from_env(username_env=\"USER\", api_key_env=\"API-KEY\")"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Please note that you pass the names of the environment variables.\n",
+ "To ensure that the above code works, you must set the environment variables with the same name you specified in `from_env`.\n",
+ "\n",
+ "Having the `config` ready, we can now initialize the actual client."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "client = StreamPipesClient(client_config=config)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "That's already it. You can check if everything works out by using the following command:"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "client.describe()"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "This prints you a short textual description of the connected StreamPipes instance to the console.\n",
+ "\n",
+ "<br>\n",
+ "\n",
+ "The created `client` instance serves as the central point of interaction for StreamPipes.\n",
+ "You can invoke a variety of commands directly on this object.\n",
+ "\n",
+ "Are you curious now how you actually can get data out of StreamPipes and make use of it with Python?\n",
+ "Then check out the next example on [extracting Data from the StreamPipes data lake]().\n",
+ "\n",
+ "<br>\n",
+ "\n",
+ "Thanks for reading this introductory example.\n",
+ "We hope you like it and would love to receive some feedback from you.\n",
+ "Just go to our [GitHub discussion page](https://github.com/apache/streampipes/discussions) and let us know your impression.\n",
+ "We'll read and react to them all, we promise!"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 2
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython2",
+ "version": "2.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/streampipes-client-python/docs/examples/2-extracting-data-from-the-streampipes-data-lake.ipynb b/streampipes-client-python/docs/examples/2-extracting-data-from-the-streampipes-data-lake.ipynb
new file mode 100644
index 000000000..558f8a710
--- /dev/null
+++ b/streampipes-client-python/docs/examples/2-extracting-data-from-the-streampipes-data-lake.ipynb
@@ -0,0 +1,391 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Extracting Data from the StreamPipes data lake\n",
+ "\n",
+ "In the first example ([Introduction to the StreamPipes Python client]()) we took the first steps with the StreamPipes Python client and learned how to set everything up.\n",
+ "Now we are ready to get started and want to retrieve some data out of StreamPipes.\n",
+ "In this tutorial, we'll focus on the StreamPipes Data Lake, the component where StreamPipes stores data internally.\n",
+ "To get started, we'll use the `client` instance created in the first tutorial."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "outputs": [],
+ "source": [
+ "from streampipes_client.client import StreamPipesClient\n",
+ "from streampipes_client.client.client_config import StreamPipesClientConfig\n",
+ "from streampipes_client.client.credential_provider import StreamPipesApiKeyCredentials"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "os.environ[\"USER\"] = \"admin@streampipes.apache.org\"\n",
+ "os.environ[\"API-KEY\"] = \"XXX\"\n"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "outputs": [],
+ "source": [
+ "config = StreamPipesClientConfig(\n",
+ " credential_provider=StreamPipesApiKeyCredentials.from_env(username_env=\"USER\", api_key_env=\"API-KEY\"),\n",
+ " host_address=\"localhost\",\n",
+ " https_disabled=True,\n",
+ " port=80\n",
+ ")"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2022-12-04 21:19:21,832 - streampipes_client.client.client - [INFO] - [client.py:127] [_set_up_logging] - Logging successfully initialized with logging level INFO.\n"
+ ]
+ }
+ ],
+ "source": [
+ "client = StreamPipesClient(client_config=config)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "As a first step, we want to get an overview about all data available in the data lake.\n",
+ "The data is stored as so-called `measures`, which refer to a data stream stored in the data lake.\n",
+ "For his purpose we use the `all()` method of the `dataLakeMeasure` endpoint."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2022-12-04 21:19:23,599 - streampipes_client.endpoint.endpoint - [INFO] - [endpoint.py:153] [_make_request] - Successfully retrieved all resources.\n"
+ ]
+ }
+ ],
+ "source": [
+ "data_lake_measures = client.dataLakeMeasureApi.all()"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "So let's see how many measures are available"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "2"
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "len(data_lake_measures)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "All resources of the StreamPipes Python client support the standard Python expressions. If not, [please let us know](https://github.com/apache/streampipes/issues/new/choose)."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "DataLakeMeasure(element_id='urn:streampipes.apache.org:spi:datalakemeasure:xLSfXZ', measure_name='test', timestamp_field='s0::timestamp', event_schema=EventSchema(element_id='urn:streampipes.apache.org:spi:eventschema:UDMHXn', event_properties=[EventProperty(element_id='urn:streampipes.apache.org:spi:eventpropertyprimitive:utvSWg', label='Density', description='Denotes the current density of the fluid', runtime_name='density', required=False, domain_properties=['http [...]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "data_lake_measures[-1]"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To ge a more comprehensive overview you can take a look at the `pandas` representation"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": " measure_name timestamp_field pipeline_id pipeline_name pipeline_is_running \\\n0 flow-rate s0::timestamp None None False \n1 test s0::timestamp None None False \n\n num_event_properties \n0 3 \n1 6 ",
+ "text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>measure_name</th>\n <th>timestamp_field</th>\n <th>pipeline_id</th>\n <th>pipeline_name</ [...]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(data_lake_measures.to_pandas())"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "So far, we have only retrieved metadata about the available data lake measure.\n",
+ "In the following, we will access the actual data of the measure `flow-rate`.\n",
+ "\n",
+ "For this purpose, we will use the `get()` method of the `dataLakeMeasure` endpoint."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2022-12-04 21:19:30,505 - streampipes_client.endpoint.endpoint - [INFO] - [endpoint.py:153] [_make_request] - Successfully retrieved all resources.\n"
+ ]
+ }
+ ],
+ "source": [
+ "flow_rate_measure = client.dataLakeMeasureApi.get(identifier=\"flow-rate\")"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "For further processing, the easiest way is to turn the data measure into a `pandas DataFrame`."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "outputs": [],
+ "source": [
+ "flow_rate_pd = flow_rate_measure.to_pandas()"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Let's see how many data points we got..."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "2020"
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "len(flow_rate_pd)"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "... and get a first overview"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": " mass_flow temperature\ncount 2020.000000 2020.000000\nmean 4.976635 52.688616\nstd 2.920448 8.756244\nmin 0.003300 40.002800\n25% 2.443325 45.250551\n50% 4.886400 50.289900\n75% 7.524550 60.050674\nmax 9.997400 69.993896",
+ "text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>mass_flow</th>\n <th>temperature</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>count< [...]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "flow_rate_pd.describe()"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "As a final step, we want to create a plot of both attributes"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "outputs": [
+ {
+ "data": {
+ "text/plain": "<Figure size 640x480 with 1 Axes>",
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC8Z0lEQVR4nOydd3gVRffHvzc9gRQIkNBB6b0oEFBBRIHXDvaO2AEFLMjvtYG+4mvF3l4FG6JYUEBAQHrvvYZOCjUJBFLv/v7Y3JvdvbOzM3v33tzA+TwPD7m7szOzu7MzZ845c8alKIoCgiAIgiCIIBFW0RUgCIIgCOLCgoQPgiAIgiCCCgkfBEEQBEEEFRI+CIIgCIIIKiR8EARBEAQRVEj4IAiCIAgiqJDwQRAEQRBEUCHhgyAIgiCIoBJR0RUw4na7kZGRgfj4eLhcroquDkEQBEEQAiiKgtOnT6NOnToIC+Pr [...]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "flow_rate_pd.plot(y=[\"mass_flow\", \"temperature\"])\n",
+ "plt.show()"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "... from this point on we leave all future processing of the data up to your creativity.\n",
+ "Keep in mind: the general syntax used in this example (`all()`, `to_pandas()`, `get()`) applies to all endpoints and associated resources of the StreamPipes Python client.\n",
+ "\n",
+ "If you get further and create exiting stuff with data extracted from StreamPipes please [let us know](https://github.com/apache/streampipes/discussions/categories/show-and-tell).\n",
+ "We are thrilled to see what you as a community will build with the provided client.\n",
+ "Furthermore, don't hesitate to discuss [feature requests](https://github.com/apache/streampipes/discussions/812) to extend the current functionality with us."
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "How do you like this tutorial?\n",
+ "We hope you like it and would love to receive some feedback from you.\n",
+ "Just go to our [GitHub discussion page](https://github.com/apache/streampipes/discussions) and let us know your impression.\n",
+ "We'll read and react to them all, we promise!"
+ ],
+ "metadata": {
+ "collapsed": false
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [],
+ "metadata": {
+ "collapsed": false
+ }
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 2
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython2",
+ "version": "2.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/streampipes-client-python/docs/getting-started/installation.md b/streampipes-client-python/docs/getting-started/installation.md
index b072c5b21..e6ddce8ed 100644
--- a/streampipes-client-python/docs/getting-started/installation.md
+++ b/streampipes-client-python/docs/getting-started/installation.md
@@ -22,7 +22,7 @@ The StreamPipes Python Client is meant to work with Python 3.8 and above. Instal
You can install the latest development version from GitHub, as so:
```bash
-pip install <NEEDS TO BE CLARIFIED>
+pip install git+https://github.com/apache/streampipes.git#subdirectory=streampipes-client-python
```
---
diff --git a/streampipes-client-python/docs/getting-started/quickstart.md b/streampipes-client-python/docs/getting-started/quickstart.md
index 5a80a9f06..9a0a2db04 100644
--- a/streampipes-client-python/docs/getting-started/quickstart.md
+++ b/streampipes-client-python/docs/getting-started/quickstart.md
@@ -19,7 +19,6 @@
# ⚡️ Quickstart
As a quick example, we demonstrate how to set up and configure a StreamPipes client.
-In addition, we will get the available data lake measures out of StreamPipes.
```python
>>> from streampipes_client.client import StreamPipesClient
@@ -37,19 +36,13 @@ In addition, we will get the available data lake measures out of StreamPipes.
...)
>>> client = StreamPipesClient(client_config=config)
+>>> client.describe()
-# get all available datat lake measures
->>> measures = client.dataLakeMeasureApi.all()
-
-# get amount of retrieved measures
->>> len(measures)
-1
-
-# inspect the data lake measures as pandas dataframe
->>> measures.to_pandas()
- measure_name timestamp_field ... pipeline_is_running num_event_properties
-0 test s0::timestamp ... False 2
-[1 rows x 6 columns]
+Hi there!
+You are connected to a StreamPipes instance running at http://localhost:80.
+The following StreamPipes resources are available with this client:
+6x DataStreams
+1x DataLakeMeasures
```
<br>
Alternatively, you can provide your credentials via environment variables.
diff --git a/streampipes-client-python/mkdocs.yml b/streampipes-client-python/mkdocs.yml
index 5bf0f9e83..bc129cea5 100644
--- a/streampipes-client-python/mkdocs.yml
+++ b/streampipes-client-python/mkdocs.yml
@@ -56,6 +56,7 @@ markdown_extensions:
plugins:
- awesome-pages
- autorefs
+ - mkdocs-jupyter
- gen-files:
scripts:
- docs/scripts/gen_ref_pages.py
@@ -78,4 +79,7 @@ nav:
- Getting Started:
- Installation: getting-started/installation.md
- Quickstart: getting-started/quickstart.md
+ - Examples:
+ - Introduction to the StreamPipes Python Client: examples/1-introduction-to-streampipes-python-client.ipynb
+ - Extracting Data from the StreamPipes Data Lake: examples/2-extracting-data-from-the-streampipes-data-lake.ipynb
- Reference: reference/*
\ No newline at end of file
diff --git a/streampipes-client-python/setup.py b/streampipes-client-python/setup.py
index 99a5ee2be..4c642c480 100644
--- a/streampipes-client-python/setup.py
+++ b/streampipes-client-python/setup.py
@@ -60,6 +60,7 @@ docs_packages = [
"mkdocs-gen-files>=0.3.5",
"mkdocs-literate-nav>=0.4.1",
"numpydoc>=1.2",
+ "mkdocs-jupyter>=0.22.0 "
]
here = os.path.abspath(os.path.dirname(__file__))
[streampipes] 01/02: add describe() method to python client
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bossenti pushed a commit to branch add-example-files-python-client
in repository https://gitbox.apache.org/repos/asf/streampipes.git
commit e3cac9dd6d742a9c08e272eb62b14b8751b1abf0
Author: bossenti <bo...@posteo.de>
AuthorDate: Sun Dec 4 21:43:35 2022 +0100
add describe() method to python client
---
.../streampipes_client/client/client.py | 37 +++++++++++++++++
.../model/resource/data_lake_measure.py | 6 +--
.../tests/client/test_client.py | 46 +++++++++++++++++++++-
.../tests/client/test_endpoint.py | 44 ++++++++++++++++++---
4 files changed, 121 insertions(+), 12 deletions(-)
diff --git a/streampipes-client-python/streampipes_client/client/client.py b/streampipes-client-python/streampipes_client/client/client.py
index 0a6a8e1c3..3fb49f28a 100644
--- a/streampipes-client-python/streampipes_client/client/client.py
+++ b/streampipes-client-python/streampipes_client/client/client.py
@@ -30,6 +30,7 @@ from typing import Dict, Optional
from requests import Session
from streampipes_client.client.client_config import StreamPipesClientConfig
from streampipes_client.endpoint import DataLakeMeasureEndpoint, DataStreamEndpoint
+from streampipes_client.endpoint.endpoint import APIEndpoint
logger = logging.getLogger(__name__)
@@ -176,3 +177,39 @@ class StreamPipesClient:
f"{self.client_config.host_address}:"
f"{self.client_config.port}/streampipes-backend/"
)
+
+ def describe(self) -> None:
+ """Prints short description of the connected StreamPipes instance and the available resources to the console.
+
+ Returns
+ -------
+ None
+ """
+
+ # get all endpoints of this client
+ available_endpoints = [
+ attr_name for attr_name in dir(self) if isinstance(self.__getattribute__(attr_name), APIEndpoint)
+ ]
+
+ # collect the number of available resources per endpoint
+ endpoint_stats = {
+ (all_items := self.__getattribute__(endpoint_name).all()).__class__.__name__: len(all_items)
+ for endpoint_name in available_endpoints
+ }
+
+ # sort the endpoints descending based on the number of resources
+ sorted_endpoint_stats = {
+ key: val for key, val in sorted(endpoint_stats.items(), key=lambda item: item[1], reverse=True)
+ }
+
+ base_message = (
+ f"\nHi there!\n"
+ f"You are connected to a StreamPipes instance running at "
+ f"{'http://' if self.client_config.https_disabled else 'https://'}"
+ f"{self.client_config.host_address}:{self.client_config.port}.\n"
+ f"The following StreamPipes resources are available with this client:\n"
+ )
+
+ endpoint_stats_message = "\n".join(f"{count}x {name}" for name, count in sorted_endpoint_stats.items())
+
+ print(base_message + endpoint_stats_message)
diff --git a/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py b/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
index e588267d0..9d2b73b27 100644
--- a/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
+++ b/streampipes-client-python/streampipes_client/model/resource/data_lake_measure.py
@@ -20,10 +20,6 @@ from pydantic import StrictBool, StrictStr
from streampipes_client.model.common import EventSchema
from streampipes_client.model.resource.resource import Resource
-"""
-Implementation of a resource for a data lake measure.
-"""
-
__all__ = [
"DataLakeMeasure",
]
@@ -51,7 +47,7 @@ class DataLakeMeasure(Resource):
measure_name: StrictStr
timestamp_field: StrictStr
- event_schema: EventSchema
+ event_schema: Optional[EventSchema]
pipeline_id: Optional[StrictStr]
pipeline_name: Optional[StrictStr]
pipeline_is_running: StrictBool
diff --git a/streampipes-client-python/tests/client/test_client.py b/streampipes-client-python/tests/client/test_client.py
index 0c4027647..5c53944da 100644
--- a/streampipes-client-python/tests/client/test_client.py
+++ b/streampipes-client-python/tests/client/test_client.py
@@ -14,8 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
+import json
+from collections import namedtuple
from unittest import TestCase
+from unittest.mock import MagicMock, call, patch
from streampipes_client.client import StreamPipesClient
from streampipes_client.client.client_config import StreamPipesClientConfig
@@ -68,3 +70,45 @@ class TestStreamPipesClient(TestCase):
)
self.assertTrue(isinstance(result.dataLakeMeasureApi, DataLakeMeasureEndpoint))
self.assertEqual(result.base_api_path, "https://localhost:443/streampipes-backend/")
+
+ @patch("builtins.print")
+ @patch("streampipes_client.endpoint.endpoint.APIEndpoint._make_request", autospec=True)
+ def test_client_describe(self, make_request: MagicMock, mocked_print: MagicMock):
+ def simulate_response(*args, **kwargs):
+ Response = namedtuple("Response", ["text"])
+ if "measurements" in kwargs["url"]:
+ return Response(
+ json.dumps(
+ [
+ {
+ "measureName": "test",
+ "timestampField": "time",
+ "pipelineIsRunning": False,
+ "schemaVersion": "0",
+ }
+ ]
+ )
+ )
+ if "streams" in kwargs["url"]:
+ return Response(json.dumps([{"name": "test"}]))
+
+ make_request.side_effect = simulate_response
+ StreamPipesClient.create(
+ client_config=StreamPipesClientConfig(
+ credential_provider=StreamPipesApiKeyCredentials(username="user", api_key="key"),
+ host_address="localhost",
+ https_disabled=False,
+ port=443,
+ )
+ ).describe()
+
+ mocked_print.assert_has_calls(
+ calls=[
+ call(
+ "\nHi there!\nYou are connected to a StreamPipes instance running at https://localhost:443.\n"
+ "The following StreamPipes resources are available with this client:\n"
+ "1x DataLakeMeasures\n1x DataStreams"
+ ),
+ ],
+ any_order=True,
+ )
diff --git a/streampipes-client-python/tests/client/test_endpoint.py b/streampipes-client-python/tests/client/test_endpoint.py
index d2f60409d..1117634d5 100644
--- a/streampipes-client-python/tests/client/test_endpoint.py
+++ b/streampipes-client-python/tests/client/test_endpoint.py
@@ -17,8 +17,9 @@
import json
from copy import deepcopy
+from typing import Dict, List
from unittest import TestCase
-from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, call, patch
from pydantic import ValidationError
from requests import HTTPError
@@ -30,6 +31,7 @@ from streampipes_client.model.container.resource_container import (
StreamPipesDataModelError,
StreamPipesResourceContainerJSONError,
)
+from streampipes_client.model.resource import DataStream
class TestStreamPipesEndpoints(TestCase):
@@ -89,7 +91,7 @@ class TestStreamPipesEndpoints(TestCase):
}
]
- self.data_stream_all = [
+ self.data_stream_all: List[Dict] = [
{
"elementId": "urn:streampipes.apache.org:eventstream:uPDKLI",
"name": "Test",
@@ -111,7 +113,8 @@ class TestStreamPipesEndpoints(TestCase):
"brokerHostname": "nats",
"topicDefinition": {
"elementId": "urn:streampipes.apache.org:spi:simpletopicdefinition:QzCiFI",
- "actualTopicName": "org.apache.streampipes.connect.fc22b8f6-698a-4127-aa71-e11854dc57c5",
+ "actualTopicName": "org.apache.streampipes.connect."
+ "fc22b8f6-698a-4127-aa71-e11854dc57c5",
},
"port": 4222,
}
@@ -168,7 +171,8 @@ class TestStreamPipesEndpoints(TestCase):
"measurementCapability": None,
"measurementObject": None,
"index": 0,
- "correspondingAdapterId": "urn:streampipes.apache.org:spi:org.apache.streampipes.connect.iiot.adapters.simulator.machine:11934d37-135b-4ef6-b5f1-4f520cc81a43",
+ "correspondingAdapterId": "urn:streampipes.apache.org:spi:org.apache.streampipes.connect."
+ "iiot.adapters.simulator.machine:11934d37-135b-4ef6-b5f1-4f520cc81a43",
"category": None,
"uri": "urn:streampipes.apache.org:eventstream:uPDKLI",
"dom": None,
@@ -176,6 +180,7 @@ class TestStreamPipesEndpoints(TestCase):
]
self.data_stream_all_json = json.dumps(self.data_stream_all)
+ self.data_stream_get = self.data_stream_all[0]
self.data_lake_measure_all_json = json.dumps(self.data_lake_measure_all)
self.data_lake_measure_all_json_error = json.dumps(self.data_lake_measure_all[0])
@@ -183,6 +188,33 @@ class TestStreamPipesEndpoints(TestCase):
self.dlm_all_manipulated[0]["measureName"] = False
self.data_lake_measure_all_json_validation = json.dumps(self.dlm_all_manipulated)
+ @patch("streampipes_client.client.client.Session", autospec=True)
+ def test_endpoint_get(self, http_session: MagicMock):
+ http_session_mock = MagicMock()
+ http_session_mock.get.return_value.json.return_value = self.data_stream_get
+ http_session.return_value = http_session_mock
+
+ client = StreamPipesClient(
+ client_config=StreamPipesClientConfig(
+ credential_provider=StreamPipesApiKeyCredentials(username="user", api_key="key"),
+ host_address="localhost",
+ )
+ )
+
+ result = client.dataStreamApi.get(identifier="urn:streampipes.apache.org:eventstream:uPDKLI")
+
+ http_session.assert_has_calls(
+ calls=[
+ call().get(
+ url="https://localhost:80/streampipes-backend/api/v2/streams/urn:streampipes."
+ "apache.org:eventstream:uPDKLI"
+ )
+ ],
+ any_order=True,
+ )
+ self.assertTrue(isinstance(result, DataStream))
+ self.assertEqual(result.dict(by_alias=True), self.data_stream_get)
+
@patch("streampipes_client.client.client.Session", autospec=True)
def test_endpoint_data_stream_happy_path(self, http_session: MagicMock):
http_session_mock = MagicMock()
@@ -205,7 +237,7 @@ class TestStreamPipesEndpoints(TestCase):
)
self.assertEqual(
"Test",
- result[0].name,
+ result[0].name, # type: ignore
)
self.assertEqual(
self.data_stream_all_json,
@@ -268,7 +300,7 @@ class TestStreamPipesEndpoints(TestCase):
)
self.assertEqual(
"test",
- result[0].measure_name,
+ result[0].measure_name, # type: ignore
)
self.assertEqual(
self.data_lake_measure_all_json,