You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by te...@apache.org on 2023/03/21 18:37:48 UTC
[druid] branch master updated: pip install for Python Druid API (#13938)
This is an automated email from the ASF dual-hosted git repository.
techdocsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new ede9903ff4 pip install for Python Druid API (#13938)
ede9903ff4 is described below
commit ede9903ff4752f643716ad82bb20ecfe60a559f9
Author: Victoria Lim <vt...@users.noreply.github.com>
AuthorDate: Tue Mar 21 11:37:39 2023 -0700
pip install for Python Druid API (#13938)
Broken test appears unrelated to this PR
* make druidapi pip installable
* include druidapi in prerequisites
* add license to setup.py
* updates from Paul's review
* note about editable install
* Apply suggestions from code review
Co-authored-by: 317brian <53...@users.noreply.github.com>
* update install instructions
* found unrelated typos
* standardize install cmd with pip
---------
Co-authored-by: 317brian <53...@users.noreply.github.com>
---
.gitignore | 3 +
.../{-START HERE-.ipynb => 0-START-HERE.ipynb} | 2 +-
.../jupyter-notebooks/Python_API_Tutorial.ipynb | 2 +-
examples/quickstart/jupyter-notebooks/README.md | 23 +++--
.../jupyter-notebooks/api-tutorial.ipynb | 2 +-
.../jupyter-notebooks/druidapi/README.md | 97 ++++++++++++----------
.../druidapi/{ => druidapi}/__init__.py | 5 +-
.../druidapi/{ => druidapi}/base_table.py | 0
.../druidapi/{ => druidapi}/catalog.py | 12 +--
.../druidapi/{ => druidapi}/consts.py | 0
.../druidapi/{ => druidapi}/datasource.py | 24 +++---
.../druidapi/{ => druidapi}/display.py | 8 +-
.../druidapi/{ => druidapi}/druid.py | 22 ++---
.../druidapi/{ => druidapi}/error.py | 0
.../druidapi/{html.py => druidapi/html_display.py} | 8 +-
.../druidapi/{ => druidapi}/rest.py | 18 ++--
.../druidapi/{ => druidapi}/sql.py | 24 +++---
.../druidapi/{ => druidapi}/status.py | 0
.../druidapi/{ => druidapi}/tasks.py | 12 +--
.../druidapi/{text.py => druidapi/text_display.py} | 12 +--
.../druidapi/{ => druidapi}/util.py | 4 +-
.../druidapi/{util.py => setup.py} | 38 +++++----
22 files changed, 171 insertions(+), 145 deletions(-)
diff --git a/.gitignore b/.gitignore
index 30e00e2d42..5d9da29720 100644
--- a/.gitignore
+++ b/.gitignore
@@ -30,3 +30,6 @@ integration-tests/gen-scripts/
*.hprof
**/.ipynb_checkpoints/
*.pyc
+**/.ipython/
+**/.jupyter/
+**/.local/
diff --git a/examples/quickstart/jupyter-notebooks/-START HERE-.ipynb b/examples/quickstart/jupyter-notebooks/0-START-HERE.ipynb
similarity index 99%
rename from examples/quickstart/jupyter-notebooks/-START HERE-.ipynb
rename to examples/quickstart/jupyter-notebooks/0-START-HERE.ipynb
index 9c88edc896..fe4a30a551 100644
--- a/examples/quickstart/jupyter-notebooks/-START HERE-.ipynb
+++ b/examples/quickstart/jupyter-notebooks/0-START-HERE.ipynb
@@ -46,7 +46,7 @@
"- The `requests` package for Python. For example, you can install it with the following command:\n",
"\n",
" ```bash\n",
- " pip3 install requests\n",
+ " pip install requests\n",
" ````\n",
"\n",
"- JupyterLab (recommended) or Jupyter Notebook running on a non-default port. By default, Druid\n",
diff --git a/examples/quickstart/jupyter-notebooks/Python_API_Tutorial.ipynb b/examples/quickstart/jupyter-notebooks/Python_API_Tutorial.ipynb
index 51239edb43..9077d2bc28 100644
--- a/examples/quickstart/jupyter-notebooks/Python_API_Tutorial.ipynb
+++ b/examples/quickstart/jupyter-notebooks/Python_API_Tutorial.ipynb
@@ -564,7 +564,7 @@
"id": "2654e72c",
"metadata": {},
"source": [
- "Use the REST client if you need to make calls that are not yet wrapped by the Python API, or if you want to do something special. To illustrate the client, you can make some of the same calls as in the [Druid REST API notebook](api_tutorial.ipynb).\n",
+ "Use the REST client if you need to make calls that are not yet wrapped by the Python API, or if you want to do something special. To illustrate the client, you can make some of the same calls as in the [Druid REST API notebook](api-tutorial.ipynb).\n",
"\n",
"The REST API maintains the Druid host: you just provide the specifc URL tail. There are methods to get or post JSON results. For example, to get status information:"
]
diff --git a/examples/quickstart/jupyter-notebooks/README.md b/examples/quickstart/jupyter-notebooks/README.md
index 823d2136d7..826ae5df34 100644
--- a/examples/quickstart/jupyter-notebooks/README.md
+++ b/examples/quickstart/jupyter-notebooks/README.md
@@ -1,9 +1,9 @@
# Jupyter Notebook tutorials for Druid
-If you are reading this in Jupyter, switch over to the [- START HERE -](- START HERE -.ipynb]
+If you are reading this in Jupyter, switch over to the [0-START-HERE](0-START-HERE.ipynb)
notebook instead.
-<!-- This README, the "- START HERE -" notebook, and the tutorial-jupyter-index.md file in
+<!-- This README, the "0-START-HERE" notebook, and the tutorial-jupyter-index.md file in
docs/tutorials share a lot of the same content. If you make a change in one place, update
the other too. -->
@@ -39,7 +39,7 @@ Make sure you meet the following requirements before starting the Jupyter-based
- The `requests` package for Python. For example, you can install it with the following command:
```bash
- pip3 install requests
+ pip install requests
```
- JupyterLab (recommended) or Jupyter Notebook running on a non-default port. By default, Druid
@@ -49,9 +49,9 @@ Make sure you meet the following requirements before starting the Jupyter-based
```bash
# Install JupyterLab
- pip3 install jupyterlab
+ pip install jupyterlab
# Install Jupyter Notebook
- pip3 install notebook
+ pip install notebook
```
- Start Jupyter using either JupyterLab
```bash
@@ -65,8 +65,15 @@ Make sure you meet the following requirements before starting the Jupyter-based
jupyter notebook --port 3001
```
-- An available Druid instance. You can use the `micro-quickstart` configuration
- described in [Quickstart](https://druid.apache.org/docs/latest/tutorials/index.html).
+- The Python API client for Druid. Clone the Druid repo if you haven't already.
+Go to your Druid source repo and install `druidapi` with the following commands:
+
+ ```bash
+ cd examples/quickstart/jupyter-notebooks/druidapi
+ pip install .
+ ```
+
+- An available Druid instance. You can use the [quickstart deployment](https://druid.apache.org/docs/latest/tutorials/index.html).
The tutorials assume that you are using the quickstart, so no authentication or authorization
is expected unless explicitly mentioned.
@@ -85,4 +92,4 @@ Make sure you meet the following requirements before starting the Jupyter-based
## Continue in Jupyter
-Start Jupyter (see above) and navigate to the "- START HERE -" page for more information.
+Start Jupyter (see above) and navigate to the "0-START-HERE" notebook for more information.
diff --git a/examples/quickstart/jupyter-notebooks/api-tutorial.ipynb b/examples/quickstart/jupyter-notebooks/api-tutorial.ipynb
index 7bb7224662..11f3bdf9ba 100644
--- a/examples/quickstart/jupyter-notebooks/api-tutorial.ipynb
+++ b/examples/quickstart/jupyter-notebooks/api-tutorial.ipynb
@@ -63,7 +63,7 @@
"Install the [Requests](https://requests.readthedocs.io/en/latest/) library for Python before you start. For example:\n",
"\n",
"```bash\n",
- "pip3 install requests\n",
+ "pip install requests\n",
"```\n",
"\n",
"Please read the [Requests Quickstart](https://requests.readthedocs.io/en/latest/user/quickstart/) to gain a basic understanding of how Requests works.\n",
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/README.md b/examples/quickstart/jupyter-notebooks/druidapi/README.md
index 0c9218a904..d4901b3a3a 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/README.md
+++ b/examples/quickstart/jupyter-notebooks/druidapi/README.md
@@ -28,6 +28,9 @@ in any Python environment, but is optimized for use in Jupyter, providing a comp
environment which complements the UI-based Druid console. The primary use of `druidapi` at present
is to support the set of tutorial notebooks provided in the parent directory.
+`druidapi` works against any version of Druid. Operations that make use of newer features obviously work
+only against versions of Druid that support those features.
+
## Install
At present, the best way to use `druidapi` is to clone the Druid repo itself:
@@ -36,21 +39,29 @@ At present, the best way to use `druidapi` is to clone the Druid repo itself:
git clone git@github.com:apache/druid.git
```
-`druidapi` is located in `examples/quickstart/jupyter-notebooks/druidapi/`
+`druidapi` is located in `examples/quickstart/jupyter-notebooks/druidapi/`.
+From this directory, install the package and its dependencies with pip using the following command:
+
+```
+pip install .
+```
-Eventually we would like to create a Python package that can be installed with `pip`. Contributions
-in that area are welcome.
+Note that there is a second level `druidapi` directory that contains the modules. Do not run
+the install command in the subdirectory.
-Dependencies are listed in `requirements.txt`.
+Verify your installation by checking that the following command runs in Python:
-`druidapi` works against any version of Druid. Operations that exploit newer features obviously work
-only against versions of Druid that support those features.
+```python
+import druidapi
+```
+
+The import statement should not return anything if it runs successfully.
-## Getting Started
+## Getting started
To use `druidapi`, first import the library, then connect to your cluster by providing the URL to your Router instance. The way that is done differs a bit between consumers.
-### From a Tutorial Jupyter Notebook
+### From a tutorial Jupyter notebook
The tutorial Jupyter notebooks in `examples/quickstart/jupyter-notebooks` reside in the same directory tree
as this library. We start the library using the Jupyter-oriented API which is able to render tables in
@@ -70,40 +81,17 @@ druid = druidapi.jupyter_client(router_endpoint)
The `jupyter_client` call defines a number of CSS styles to aid in displaying tabular results. It also
provides a "display" client that renders information as HTML tables.
-### From Any Other Juypter Notebook
-
-If you create a Jupyter notebook outside of the `jupyter-notebooks` directory then you must tell Python where
-to find the `druidapi` library. (This step is temporary until `druidapi` is properly packaged.)
-
-First, set a variable to point to the location where you cloned the Druid git repo:
-
-```python
-druid_dev = '/path/to/Druid-repo'
-```
-
-Then, add the notebooks directory to Python's module search path:
-
-```python
-import sys
-sys.path.append(druid_dev + '/examples/quickstart/jupyter-notebooks/')
-```
-
-Now you can import `druidapi` and create a client as shown in the previous section.
-
-### From a Python Script
+### From a Python script
`druidapi` works in any Python script. When run outside of a Jupyter notebook, the various "display"
commands revert to displaying a text (not HTML) format. The steps are similar to those above:
```python
-druid_dev = '/path/to/Druid-repo'
-import sys
-sys.path.append(druid_dev + '/examples/quickstart/jupyter-notebooks/')
import druidapi
druid = druidapi.client(router_endpoint)
```
-## Library Organization
+## Library organization
`druidapi` organizes Druid REST operations into various "clients," each of which provides operations
for one of Druid's functional areas. Obtain a client from the `druid` client created above. For
@@ -127,7 +115,7 @@ available as properties on the `druid` object created above.
* `display` - A set of convenience operations to display results as lightly formatted tables
in either HTML (for Jupyter notebooks) or text (for other Python scripts).
-## Assumed Cluster Architecture
+## Assumed cluster architecture
`druidapi` assumes that you run a standard Druid cluster with a Router in front of the other nodes.
This design works well for most Druid clusters:
@@ -148,7 +136,7 @@ The one exception to this rule is if you want to perform a health check (i.e. th
on a service other than the Router. These checks are _not_ proxied by the Router: you must connect to
the target service directly.
-## Status Operations
+## Status operations
When working with tutorials, a local Druid cluster, or a Druid integration test cluster, it is common
to start your cluster then immediately start performing `druidapi` operations. However, because Druid
@@ -183,7 +171,7 @@ extension is loaded:
status_client.properties['druid.extensions.loadList']
```
-## Display Client
+## Display client
When run in a Jupyter notebook, it is often handy to format results for display. A special display
client performs operations _and_ formats them for display as HTML tables within the notebook.
@@ -204,7 +192,7 @@ The most common methods are:
The display client also has other methods to format data as a table, to display various kinds
of messages and so on.
-## Interactive Queries
+## Interactive queries
The original [`pydruid`](https://pythonhosted.org/pydruid/) library revolves around Druid
"native" queries. Most new applications now use SQL. `druidapi` provides two ways to run
@@ -264,7 +252,7 @@ channel count
Within Jupyter, the results are formatted as an HTML table.
-### Advanced Queries
+### Advanced queries
In addition to the SQL text, Druid also lets you specify:
@@ -350,7 +338,7 @@ resp.show()
In fact, the display client `sql()` method uses the `resp.show()` method internally, which in turn uses the
`rows` and `schema` properties.
-### Run a Query and Return Results
+### Run a query and return results
The above forms are handy for interactive use in a notebook. If you just need to run a query to use the results
in code, just do the following:
@@ -366,7 +354,7 @@ sql = 'SELECT * FROM {}'
rows = sql_client.sql(sql, ['myTable'])
```
-## MSQ Queries
+## MSQ queries
The SQL client can also run an MSQ query. See the `sql-tutorial.ipynb` notebook for examples. First define the
query:
@@ -408,7 +396,7 @@ while for Druid to load the resulting segments, so you must wait for the table t
sql_client.wait_until_ready('myTable')
```
-## Datasource Operations
+## Datasource operations
To get information about a datasource, prefer to query the `INFORMATION_SCHEMA` tables, or use the methods
in the display client. Use the datasource client for other operations.
@@ -425,7 +413,7 @@ datasources.drop('myWiki', True)
The True argument asks for "if exists" semantics so you don't get an error if the datasource does not exist.
-## REST Client
+## REST client
The `druidapi` is based on a simple REST client which is itself based on the Requests library. If you
need to use Druid REST APIs not yet wrapped by this library, you can use the REST client directly.
@@ -495,3 +483,28 @@ Druid has a large number of special constants: type names, options, etc. The con
from druidapi import consts
help(consts)
```
+
+## Contributing
+
+We encourage you to contribute to the `druidapi` package.
+Set up an editable installation for development by running the following command
+in a local clone of your `apache/druid` repo in
+`examples/quickstart/jupyter-notebooks/druidapi/`:
+
+```
+pip install -e .
+```
+
+An editable installation allows you to implement and test changes iteratively
+without having to reinstall the package with every change.
+
+When you update the package, also increment the version field in `setup.py` following the
+[PEP 440 semantic versioning scheme](https://peps.python.org/pep-0440/#semantic-versioning).
+
+Use the following guidelines for incrementing the version number:
+* Increment the third position for a patch or bug fix.
+* Increment the second position for new features, such as adding new method wrappers.
+* Increment the first position for major changes and changes that are not backwards compatible.
+
+Submit your contribution by opening a pull request to the `apache/druid` GitHub repository.
+
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/__init__.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/__init__.py
similarity index 94%
rename from examples/quickstart/jupyter-notebooks/druidapi/__init__.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/__init__.py
index 2734544ea4..a48a23d91a 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/__init__.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/__init__.py
@@ -13,14 +13,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .druid import DruidClient
+from druidapi.druid import DruidClient
def jupyter_client(endpoint) -> DruidClient:
'''
Create a Druid client configured to display results as HTML withing a Jupyter notebook.
Waits for the cluster to become ready to avoid intermitent problems when using Druid.
'''
- from .html import HtmlDisplayClient
+ from druidapi.html_display import HtmlDisplayClient
druid = DruidClient(endpoint, HtmlDisplayClient())
druid.status.wait_until_ready()
return druid
@@ -33,3 +33,4 @@ def client(endpoint) -> DruidClient:
that the cluster has not yet fully started.
'''
return DruidClient(endpoint)
+
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/base_table.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/base_table.py
similarity index 100%
rename from examples/quickstart/jupyter-notebooks/druidapi/base_table.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/base_table.py
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/catalog.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/catalog.py
similarity index 96%
rename from examples/quickstart/jupyter-notebooks/druidapi/catalog.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/catalog.py
index fb9560a9fd..a54148ef43 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/catalog.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/catalog.py
@@ -14,8 +14,8 @@
# limitations under the License.
import requests
-from .consts import COORD_BASE
-from .rest import check_error
+from druidapi.consts import COORD_BASE
+from druidapi.rest import check_error
# Catalog (new feature in Druid 26)
CATALOG_BASE = COORD_BASE + '/catalog'
@@ -30,10 +30,10 @@ class CatalogClient:
Client for the Druid catalog feature that provides metadata for tables,
including both datasources and external tables.
'''
-
+
def __init__(self, rest_client):
self.client = rest_client
-
+
def post_table(self, schema, table_name, table_spec, version=None, overwrite=None):
params = {}
if version:
@@ -44,7 +44,7 @@ class CatalogClient:
def create(self, schema, table_name, table_spec):
self.post_table(schema, table_name, table_spec)
-
+
def table(self, schema, table_name):
return self.client.get_json(REQ_CAT_SCHEMA_TABLE, args=[schema, table_name])
@@ -53,7 +53,7 @@ class CatalogClient:
if if_exists and r.status_code == requests.codes.not_found:
return
check_error(r)
-
+
def edit_table(self, schema, table_name, action):
return self.client.post_json(REQ_CAT_SCHEMA_TABLE_EDIT, action, args=[schema, table_name])
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/consts.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/consts.py
similarity index 100%
rename from examples/quickstart/jupyter-notebooks/druidapi/consts.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/consts.py
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/datasource.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/datasource.py
similarity index 89%
rename from examples/quickstart/jupyter-notebooks/druidapi/datasource.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/datasource.py
index 7a12630d10..4e0febaef4 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/datasource.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/datasource.py
@@ -14,9 +14,9 @@
# limitations under the License.
import requests, time
-from .consts import COORD_BASE
-from .rest import check_error
-from .util import dict_get
+from druidapi.consts import COORD_BASE
+from druidapi.rest import check_error
+from druidapi.util import dict_get
REQ_DATASOURCES = COORD_BASE + '/datasources'
REQ_DATASOURCE = REQ_DATASOURCES + '/{}'
@@ -32,18 +32,18 @@ class DatasourceClient:
See https://druid.apache.org/docs/latest/operations/api-reference.html#datasources
'''
-
+
def __init__(self, rest_client):
self.rest_client = rest_client
-
+
def drop(self, ds_name, if_exists=False):
'''
Drops a data source.
- Marks as unused all segments belonging to a datasource.
+ Marks as unused all segments belonging to a datasource.
Marking all segments as unused is equivalent to dropping the table.
-
+
Parameters
----------
ds_name: str
@@ -51,9 +51,9 @@ class DatasourceClient:
Returns
-------
- Returns a map of the form
- {"numChangedSegments": <number>} with the number of segments in the database whose
- state has been changed (that is, the segments were marked as unused) as the result
+ Returns a map of the form
+ {"numChangedSegments": <number>} with the number of segments in the database whose
+ state has been changed (that is, the segments were marked as unused) as the result
of this API call.
Reference
@@ -67,10 +67,10 @@ class DatasourceClient:
def load_status_req(self, ds_name, params=None):
return self.rest_client.get_json(REQ_DS_LOAD_STATUS, args=[ds_name], params=params)
-
+
def load_status(self, ds_name):
return self.load_status_req(ds_name, {
- 'forceMetadataRefresh': 'true',
+ 'forceMetadataRefresh': 'true',
'interval': '1970-01-01/2999-01-01'})
def wait_until_ready(self, ds_name):
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/display.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/display.py
similarity index 99%
rename from examples/quickstart/jupyter-notebooks/druidapi/display.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/display.py
index e4a139c1f6..5b4368325c 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/display.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/display.py
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from . import consts
+from druidapi import consts
class DisplayClient:
'''
@@ -41,7 +41,7 @@ class DisplayClient:
def new_table(self):
raise NotImplementedError()
-
+
def show_table(self, table):
raise NotImplementedError()
@@ -114,7 +114,7 @@ class DisplayClient:
def table(self, table_name):
'''
Describe a table by returning the list of columns in the table.
-
+
Parameters
----------
table_name str
@@ -122,7 +122,7 @@ class DisplayClient:
If the form is "table", then the 'druid' schema is assumed.
'''
self._druid.sql._schema_query(table_name).show(display=self)
-
+
def function(self, table_name):
'''
Retrieve the list of parameters for a partial external table defined in
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/druid.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/druid.py
similarity index 93%
rename from examples/quickstart/jupyter-notebooks/druidapi/druid.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/druid.py
index 3e0d154068..fad923bab0 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/druid.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/druid.py
@@ -13,12 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .rest import DruidRestClient
-from .status import StatusClient
-from .catalog import CatalogClient
-from .sql import QueryClient
-from .tasks import TaskClient
-from .datasource import DatasourceClient
+from druidapi.rest import DruidRestClient
+from druidapi.status import StatusClient
+from druidapi.catalog import CatalogClient
+from druidapi.sql import QueryClient
+from druidapi.tasks import TaskClient
+from druidapi.datasource import DatasourceClient
class DruidClient:
'''
@@ -36,7 +36,7 @@ class DruidClient:
if display_client:
self.display_client = display_client
else:
- from .text import TextDisplayClient
+ from druidapi.text_display import TextDisplayClient
self.display_client = TextDisplayClient()
self.display_client._druid = self
@@ -58,7 +58,7 @@ class DruidClient:
to learn what the code does so you can replicate it in your own client.
'''
self.rest_client.enable_trace(enable)
-
+
@property
def status(self) -> StatusClient:
'''
@@ -67,7 +67,7 @@ class DruidClient:
if not self.status_client:
self.status_client = StatusClient(self.rest_client)
return self.status_client
-
+
def status_for(self, endpoint) -> StatusClient:
'''
Returns the status client for a Druid service.
@@ -116,11 +116,11 @@ class DruidClient:
if not self.datasource_client:
self.datasource_client = DatasourceClient(self.rest_client)
return self.datasource_client
-
+
@property
def display(self):
return self.display_client
-
+
def close(self):
self.rest_client.close()
self.rest_client = None
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/error.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/error.py
similarity index 100%
rename from examples/quickstart/jupyter-notebooks/druidapi/error.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/error.py
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/html.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/html_display.py
similarity index 97%
rename from examples/quickstart/jupyter-notebooks/druidapi/html.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/html_display.py
index e871c2b785..e63946993f 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/html.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/html_display.py
@@ -15,8 +15,8 @@
from IPython.display import display, HTML
from html import escape
-from .display import DisplayClient
-from .base_table import BaseTable
+from druidapi.display import DisplayClient
+from druidapi.base_table import BaseTable
STYLES = '''
<style>
@@ -124,7 +124,7 @@ class HtmlDisplayClient(DisplayClient):
if not initialized:
display(HTML(STYLES))
initialized = True
-
+
def text(self, msg):
html('<div class="druid">' + escape_for_html(msg) + '</div>')
@@ -136,6 +136,6 @@ class HtmlDisplayClient(DisplayClient):
def new_table(self):
return HtmlTable()
-
+
def show_table(self, table):
self.text(table.format())
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/rest.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/rest.py
similarity index 98%
rename from examples/quickstart/jupyter-notebooks/druidapi/rest.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/rest.py
index b9d62083af..081f589466 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/rest.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/rest.py
@@ -14,9 +14,9 @@
# limitations under the License.
import requests
-from .util import dict_get
+from druidapi.util import dict_get
from urllib.parse import quote
-from .error import ClientError
+from druidapi.error import ClientError
def check_error(response):
'''
@@ -31,7 +31,7 @@ def check_error(response):
This method attempts to parse these variations. If the error response JSON
matches one of the known error formats, then raises a `ClientError` with the error
message. Otherise, raises a Requests library `HTTPError` for a generic error.
- If the response includes a JSON payload, then the it is returned in the json field
+ If the response includes a JSON payload, then the it is returned in the json field
of the `HTTPError` object so that the client can perhaps decode it.
'''
code = response.status_code
@@ -43,7 +43,7 @@ def check_error(response):
except Exception:
# If we can't get the JSON, raise a Requests error
response.raise_for_status()
-
+
# Druid JSON payload. Try to make sense of the error
msg = dict_get(json, 'errorMessage')
if not msg:
@@ -51,8 +51,8 @@ def check_error(response):
if msg:
# We have an explanation from Druid. Raise a Client exception
raise ClientError(msg)
-
- # Don't know what the Druid JSON is. Raise a Requetss exception, but
+
+ # Don't know what the Druid JSON is. Raise a Requests exception, but
# add on the JSON in the hopes that the caller can make use of it.
try:
response.raise_for_status()
@@ -191,7 +191,7 @@ class DruidRestClient:
args: array[str], default = None
Arguments to include in the relative URL to replace {} markers.
-
+
headers: dict, default = None
Additional HTTP header fields to send in the request.
@@ -225,7 +225,7 @@ class DruidRestClient:
args: array[str], default = None
Arguments to include in the relative URL to replace {} markers.
-
+
headers: dict, default = None
Additional HTTP header fields to send in the request.
@@ -255,7 +255,7 @@ class DruidRestClient:
def delete_json(self, req, args=None, params=None, headers=None):
return self.delete(req, args=args, params=params, headers=headers).json()
-
+
def close(self):
'''
Close the session. Use in scripts and tests when the system will otherwise complain
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/sql.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/sql.py
similarity index 99%
rename from examples/quickstart/jupyter-notebooks/druidapi/sql.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/sql.py
index 778f4bfd83..f38d0a994a 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/sql.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/sql.py
@@ -14,9 +14,9 @@
# limitations under the License.
import time, requests
-from . import consts
-from .util import dict_get, split_table_name
-from .error import DruidError, ClientError
+from druidapi import consts
+from druidapi.util import dict_get, split_table_name
+from druidapi.error import DruidError, ClientError
REQ_SQL = consts.ROUTER_BASE + '/sql'
REQ_SQL_TASK = REQ_SQL + '/task'
@@ -50,7 +50,7 @@ class SqlRequest:
else:
self.context.update(context)
return self
-
+
def add_context(self, key, value):
return self.with_context({key: value})
@@ -93,7 +93,7 @@ class SqlRequest:
def request_headers(self, headers):
self.headers = headers
return self
-
+
def to_common_format(self):
self.header = False
self.sql_types = False
@@ -122,7 +122,7 @@ class SqlRequest:
def run(self):
return self.query_client.sql_query(self)
-
+
def request_from_sql_query(query_client, sql_query):
try:
req = SqlRequest(query_client, sql_query['query'])
@@ -273,7 +273,7 @@ class SqlQueryResult:
@property
def _druid(self):
return self.request.query_client.druid_client
-
+
@property
def result_format(self):
return self.request.result_format()
@@ -384,7 +384,7 @@ class SqlQueryResult:
def _display(self, display):
return self._druid.display if not display else display
-
+
def show(self, non_null=False, display=None):
display = self._display(display)
if not self.ok:
@@ -469,7 +469,7 @@ class QueryTaskResult:
def _druid(self):
return self._request.query_client.druid_client
-
+
def _tasks(self):
return self._druid().tasks
@@ -613,7 +613,7 @@ class QueryTaskResult:
def _display(self, display):
return self._druid().display if not display else display
-
+
def show(self, non_null=False, display=None):
display = self._display(display)
if not self.done:
@@ -629,7 +629,7 @@ class QueryTaskResult:
display.alert('Query returned no {}rows'.format("visible " if non_null else ''))
return
display.data_table(data, [c.name for c in self.schema])
-
+
class QueryClient:
def __init__(self, druid, rest_client=None):
@@ -806,7 +806,7 @@ class QueryClient:
'''
Returns the schema of a table as an array of dictionaries of the
form {"Position": "<n>", "Name": "<name>", "Type": "<type>"}
-
+
Parameters
----------
table_name: str
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/status.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/status.py
similarity index 100%
rename from examples/quickstart/jupyter-notebooks/druidapi/status.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/status.py
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/tasks.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/tasks.py
similarity index 98%
rename from examples/quickstart/jupyter-notebooks/druidapi/tasks.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/tasks.py
index 9f5945b884..c1e5666519 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/tasks.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/tasks.py
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .consts import OVERLORD_BASE
+from druidapi.consts import OVERLORD_BASE
REQ_TASKS = OVERLORD_BASE + '/tasks'
REQ_POST_TASK = OVERLORD_BASE + '/task'
@@ -29,7 +29,7 @@ class TaskClient:
See https://druid.apache.org/docs/latest/operations/api-reference.html#tasks
'''
-
+
def __init__(self, rest_client):
self.client = rest_client
@@ -40,7 +40,7 @@ class TaskClient:
Parameters
----------
state: str, default = None
- Filter list of tasks by task state. Valid options are "running",
+ Filter list of tasks by task state. Valid options are "running",
"complete", "waiting", and "pending". Constants are defined for
each of these in the `consts` file.
@@ -134,7 +134,7 @@ class TaskClient:
def submit_task(self, payload):
'''
Submits a task to the Overlord.
-
+
Returns the `taskId` of the submitted task.
Parameters
@@ -164,7 +164,7 @@ class TaskClient:
Returns
-------
The REST response.
-
+
Reference
---------
`POST /druid/indexer/v1/task/{taskId}/shutdown`
@@ -183,7 +183,7 @@ class TaskClient:
Returns
-------
The REST response.
-
+
Reference
---------
`POST /druid/indexer/v1/datasources/{dataSource}/shutdownAllTasks`
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/text.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/text_display.py
similarity index 98%
rename from examples/quickstart/jupyter-notebooks/druidapi/text.py
rename to examples/quickstart/jupyter-notebooks/druidapi/druidapi/text_display.py
index c8f1f4d907..45a9df2c6a 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/text.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/text_display.py
@@ -13,8 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .display import DisplayClient
-from .base_table import pad, BaseTable
+from druidapi.display import DisplayClient
+from druidapi.base_table import pad, BaseTable
alignments = ['', '^', '>']
@@ -130,13 +130,13 @@ class TextTable(BaseTable):
self._rows = []
table_rows = self.formatter(self.compute_def(self._rows))
return '\n'.join(table_rows)
-
+
def format_rows(self, rows, min_width, max_width):
if not self._col_fmt:
return self.default_row_format(rows, min_width, max_width)
else:
return self.apply_row_formats(rows, max_width)
-
+
def default_row_format(self, rows, min_width, max_width):
new_rows = []
if min_width <= max_width:
@@ -164,7 +164,7 @@ class TextDisplayClient(DisplayClient):
def __init__(self):
DisplayClient.__init__(self)
-
+
def text(self, msg):
print(msg)
@@ -176,6 +176,6 @@ class TextDisplayClient(DisplayClient):
def new_table(self):
return TextTable()
-
+
def show_table(self, table):
print(table.format())
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/util.py b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/util.py
similarity index 95%
copy from examples/quickstart/jupyter-notebooks/druidapi/util.py
copy to examples/quickstart/jupyter-notebooks/druidapi/druidapi/util.py
index ee6ed7de43..e2d93dad2b 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/util.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/druidapi/util.py
@@ -13,12 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .error import ClientError
+from druidapi.error import ClientError
def dict_get(dict, key, default=None):
'''
Returns the value of key in the given dict, or the default value if
- the key is not found.
+ the key is not found.
'''
if not dict:
return default
diff --git a/examples/quickstart/jupyter-notebooks/druidapi/util.py b/examples/quickstart/jupyter-notebooks/druidapi/setup.py
similarity index 51%
rename from examples/quickstart/jupyter-notebooks/druidapi/util.py
rename to examples/quickstart/jupyter-notebooks/druidapi/setup.py
index ee6ed7de43..29841b2b90 100644
--- a/examples/quickstart/jupyter-notebooks/druidapi/util.py
+++ b/examples/quickstart/jupyter-notebooks/druidapi/setup.py
@@ -13,23 +13,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-from .error import ClientError
+from setuptools import setup, find_packages
-def dict_get(dict, key, default=None):
- '''
- Returns the value of key in the given dict, or the default value if
- the key is not found.
- '''
- if not dict:
- return default
- return dict.get(key, default)
+setup(
+ name='druidapi',
+ version='0.1.0',
+ description='Python API client for Apache Druid',
+ url='https://github.com/apache/druid/tree/master/examples/quickstart/jupyter-notebooks/druidapi',
+ author='Apache Druid project',
+ author_email='dev@druid.apache.org',
+ license='Apache License 2.0',
+ packages=find_packages(),
+ install_requires=['requests'],
-def split_table_name(table_name, default_schema):
- if not table_name:
- raise ClientError('Table name is required')
- parts = table_name.split('.')
- if len(parts) > 2:
- raise ClientError('Druid supports one or two-part table names')
- if len(parts) == 2:
- return parts
- return [default_schema, parts[0]]
+ classifiers=[
+ 'Development Status :: 3 - Alpha',
+ 'Intended Audience :: Developers',
+ 'Intended Audience :: End Users/Desktop',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Operating System :: OS Independent',
+ 'Programming Language :: Python :: 3',
+ ],
+)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org