You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sdap.apache.org by sk...@apache.org on 2022/08/22 15:32:57 UTC
[incubator-sdap-nexus] branch proxy-bugfix-apidocs created (now b2bcc65)
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a change to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
at b2bcc65 Updated changelog
This branch includes the following new commits:
new 32784eb Fixed apidocs bug
new 1d5f43d Merge branch 'master' of github.com:apache/incubator-sdap-nexus into proxy-bugfix-apidocs
new c226913 Addressed bug introduced by proxy
new 2bab8cb Updated matchup output fields. Validate matchup 'parameter' param against insitu schema
new 66380a2 Updated openapi spec to support domsresults endpoint
new b2bcc65 Updated changelog
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-sdap-nexus] 03/06: Addressed bug introduced by proxy
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit c226913cd2dec45efc39c6dad2cc3354e1dee02c
Author: skorper <st...@gmail.com>
AuthorDate: Wed Aug 10 16:46:55 2022 -0700
Addressed bug introduced by proxy
---
analysis/webservice/algorithms/doms/insitu.py | 4 ++--
analysis/webservice/algorithms/doms/subsetter.py | 2 +-
analysis/webservice/algorithms_spark/Matchup.py | 4 +++-
analysis/webservice/apidocs/openapi.yml | 4 ++--
4 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/analysis/webservice/algorithms/doms/insitu.py b/analysis/webservice/algorithms/doms/insitu.py
index 6aec6ec..6368a03 100644
--- a/analysis/webservice/algorithms/doms/insitu.py
+++ b/analysis/webservice/algorithms/doms/insitu.py
@@ -20,7 +20,7 @@ def query_insitu_schema():
def query_insitu(dataset, variable, start_time, end_time, bbox, platform, depth_min, depth_max,
- items_per_page=1000, session=None):
+ items_per_page=20000, session=None):
"""
Query insitu API, page through results, and aggregate
"""
@@ -63,7 +63,7 @@ def query_insitu(dataset, variable, start_time, end_time, bbox, platform, depth_
else:
response = requests.get(next_page_url, params=params)
- logging.debug(f'Insitu request {response.url}')
+ logging.info(f'Insitu request {response.url}')
response.raise_for_status()
insitu_page_response = response.json()
diff --git a/analysis/webservice/algorithms/doms/subsetter.py b/analysis/webservice/algorithms/doms/subsetter.py
index 12021b6..3f625d0 100644
--- a/analysis/webservice/algorithms/doms/subsetter.py
+++ b/analysis/webservice/algorithms/doms/subsetter.py
@@ -94,7 +94,7 @@ class DomsResultsRetrievalHandler(BaseDomsHandler.BaseDomsQueryCalcHandler):
}
singleton = True
- def __init__(self, tile_service_factory):
+ def __init__(self, tile_service_factory, **kwargs):
BaseDomsHandler.BaseDomsQueryCalcHandler.__init__(self, tile_service_factory)
self.log = logging.getLogger(__name__)
diff --git a/analysis/webservice/algorithms_spark/Matchup.py b/analysis/webservice/algorithms_spark/Matchup.py
index 5ae74ea..b85c7b4 100644
--- a/analysis/webservice/algorithms_spark/Matchup.py
+++ b/analysis/webservice/algorithms_spark/Matchup.py
@@ -279,7 +279,9 @@ class Matchup(NexusCalcSparkHandler):
threading.Thread(target=do_result_insert).start()
# Get only the first "result_size_limit" results
- matches = matches[0:result_size_limit]
+ # '0' means returns everything
+ if result_size_limit > 0:
+ matches = matches[0:result_size_limit]
result = DomsQueryResults(results=matches, args=args,
details=details, bounds=None,
diff --git a/analysis/webservice/apidocs/openapi.yml b/analysis/webservice/apidocs/openapi.yml
index be5e294..5e288b2 100644
--- a/analysis/webservice/apidocs/openapi.yml
+++ b/analysis/webservice/apidocs/openapi.yml
@@ -78,11 +78,11 @@ paths:
example: -45,15,-30,30
- in: query
name: platforms
- description: Platforms to include for matchup consideration
+ description: Platforms to include for matchup consideration. Platform depends on which insitu dataset is selected. See `/get_cdms_schema` endpoint for a list of valid platform ids.
required: true
schema:
type: string
- example: 1,2,3,4,5,6,7,8,9
+ example: 30
- in: query
name: depthMin
description: |
[incubator-sdap-nexus] 05/06: Updated openapi spec to support domsresults endpoint
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit 66380a29667386a750fda02091e21b2d24f5d768
Author: skorper <st...@gmail.com>
AuthorDate: Mon Aug 22 08:32:16 2022 -0700
Updated openapi spec to support domsresults endpoint
---
analysis/webservice/apidocs/openapi.yml | 55 +++++++++++++++++++++++++++++++--
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/analysis/webservice/apidocs/openapi.yml b/analysis/webservice/apidocs/openapi.yml
index 5e288b2..18de2f2 100644
--- a/analysis/webservice/apidocs/openapi.yml
+++ b/analysis/webservice/apidocs/openapi.yml
@@ -123,11 +123,13 @@ paths:
- in: query
name: parameter
description: |
- The parameter of interest used for the match up.
+ The parameter of interest used for the match up. This is
+ only used for insitu measurements. See `/cdmsschema` endpoint
+ for a list of valid options.
required: false
schema:
type: string
- default: sst
+ default: sea_surface_temperature
example: sst
- in: query
name: matchOnce
@@ -687,6 +689,55 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/Error'
+ /domsresults:
+ get:
+ summary: |
+ Convert matchup result
+ operationId: domsresults
+ tags:
+ - Matchup
+ parameters:
+ - in: query
+ name: id
+ description: |
+ The ID of the matchup execution. This comes from the
+ `executionId` field in matchup output.
+ required: true
+ schema:
+ type: string
+ example: dfc907ce-b8b4-453a-b0b6-d295572ba15e
+ - in: query
+ name: output
+ description: |
+ Format for output results.
+ required: true
+ schema:
+ type: string
+ enum: ['CSV', 'NETCDF']
+ example: CSV
+ responses:
+ '200':
+ description: Successful operation
+ content:
+ text/csv:
+ schema:
+ type: string
+ application/octet-stream:
+ schema:
+ type: string
+ format: binary
+ '400':
+ description: Bad request
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ '500':
+ description: Server error
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
/stats:
get:
summary: |
[incubator-sdap-nexus] 06/06: Updated changelog
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit b2bcc653ebbf3e507c93b38654828eee172dc23b
Author: skorper <st...@gmail.com>
AuthorDate: Mon Aug 22 08:32:45 2022 -0700
Updated changelog
---
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3d97eef..35fa5c5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- SDAP-390: Updated NetCDF reader tool for data matchup and added user functionality.
- SDAP-396: Added saildrone insitu api to matchup
- Matchup validates insitu parameter using insitu API schema endpoint
+- Added domsresults endpoint to openapi spec
### Changed
-SDAP-390: Changed `/doms` to `/cdms` and `doms_reader.py` to `cdms_reader.py`
- domslist endpoint points to AWS insitu instead of doms insitu
[incubator-sdap-nexus] 01/06: Fixed apidocs bug
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit 32784eb954f738811a70224605f0c00e12c86f52
Author: skorper <st...@gmail.com>
AuthorDate: Tue Aug 9 14:11:00 2022 -0700
Fixed apidocs bug
---
analysis/setup.py | 10 +++-
.../webservice/algorithms/doms/DatasetListQuery.py | 2 +-
analysis/webservice/apidocs/__init__.py | 0
.../nexus_tornado/app_builders/NexusAppBuilder.py | 3 +-
analysis/webservice/pyproject.toml | 70 ++++++++++++++++++++++
analysis/webservice/redirect/RedirectHandler.py | 3 +-
6 files changed, 82 insertions(+), 6 deletions(-)
diff --git a/analysis/setup.py b/analysis/setup.py
index 663d9bd..0897ca2 100644
--- a/analysis/setup.py
+++ b/analysis/setup.py
@@ -50,8 +50,14 @@ setuptools.setup(
# 'webservice.nexus_tornado.request.renderers'
#],
package_data={
- 'webservice': ['config/web.ini', 'config/algorithms.ini'],
- 'webservice.algorithms.doms': ['domsconfig.ini.default']
+ 'webservice': [
+ 'config/web.ini',
+ 'config/algorithms.ini',
+ 'apidocs/index.html',
+ 'apidocs/openapi.yml',
+ 'apidocs/dataset-populate.js'
+ ],
+ 'webservice.algorithms.doms': ['domsconfig.ini.default'],
},
data_files=[
('static', ['static/index.html'])
diff --git a/analysis/webservice/algorithms/doms/DatasetListQuery.py b/analysis/webservice/algorithms/doms/DatasetListQuery.py
index ed0ca7e..79a89bd 100644
--- a/analysis/webservice/algorithms/doms/DatasetListQuery.py
+++ b/analysis/webservice/algorithms/doms/DatasetListQuery.py
@@ -34,7 +34,7 @@ class DomsDatasetListQueryHandler(BaseDomsHandler.BaseDomsQueryCalcHandler):
params = {}
singleton = True
- def __init__(self, tile_service_factory):
+ def __init__(self, tile_service_factory, **kwargs):
BaseHandler.__init__(self, tile_service_factory=tile_service_factory)
def getFacetsForInsituSource(self, source):
diff --git a/analysis/webservice/apidocs/__init__.py b/analysis/webservice/apidocs/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/analysis/webservice/nexus_tornado/app_builders/NexusAppBuilder.py b/analysis/webservice/nexus_tornado/app_builders/NexusAppBuilder.py
index 80f1c92..7b7f0d3 100644
--- a/analysis/webservice/nexus_tornado/app_builders/NexusAppBuilder.py
+++ b/analysis/webservice/nexus_tornado/app_builders/NexusAppBuilder.py
@@ -1,5 +1,4 @@
import logging
-from pathlib import Path
import importlib
from functools import partial
import pkg_resources
@@ -24,7 +23,7 @@ class NexusAppBuilder:
self.handlers.append(
(r'/apidocs', tornado.web.RedirectHandler, {"url": "/apidocs/"}))
- apidocs_path = Path(__file__).parent.parent.joinpath('apidocs').resolve()
+ apidocs_path = pkg_resources.resource_filename('webservice.apidocs', '')
self.handlers.append(
(
r'/apidocs/(.*)', tornado.web.StaticFileHandler,
diff --git a/analysis/webservice/pyproject.toml b/analysis/webservice/pyproject.toml
new file mode 100644
index 0000000..0902c98
--- /dev/null
+++ b/analysis/webservice/pyproject.toml
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[tool.poetry]
+name = "nexusanalysis"
+version = "1.6.0"
+description = "SDAP Nexus Analysis API"
+authors = ["podaac-tva <po...@jpl.nasa.gov>"]
+license = "Apache-2.0"
+readme = "README.md"
+repository = "https://github.com/podaac/l2ss-py"
+classifiers = [
+ "Topic :: Software Development :: Libraries :: Python Modules"
+]
+exclude = ['contrib', 'docs', 'tests']
+packages = [
+ { include = "podaac" },
+]
+
+[tool.poetry.dependencies]
+python = "^3.8"
+netCDF4 = "^1.5"
+xarray = {version = "*", extras = ["parallel"]}
+geopandas ="^0.10"
+Shapely = "^1.7.1"
+harmony-service-lib = { version = "^1.0.20", optional = true }
+pystac = { version = "^0.5.3", optional = true }
+julian = "^0.14"
+importlib-metadata = "^4.8.1"
+h5py = "^3.6.0"
+cf-xarray = "*"
+
+[tool.poetry.dev-dependencies]
+pytest = "^5.2"
+flake8 = "^3.7"
+pytest-cov = "^2.8"
+pylint = "^2.4"
+sphinx = "^4.4"
+pytest-benchmark = "^3.2.3"
+moto = "1.3.14"
+jsonschema = "^3.2.0"
+m2r2 = "^0.3.1"
+sphinx-rtd-theme = "^0.5.2"
+
+[tool.pytest.ini_options]
+junit_family = "xunit2"
+
+[tool.poetry.extras]
+harmony = ["harmony-service-lib", "pystac"]
+
+[tool.poetry.scripts]
+run_worker = 'podaac.subsetter.subset_worker:main'
+l2ss_harmony = 'podaac.subsetter.subset_harmony:main'
+l2ss-py = 'podaac.subsetter.run_subsetter:main'
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
\ No newline at end of file
diff --git a/analysis/webservice/redirect/RedirectHandler.py b/analysis/webservice/redirect/RedirectHandler.py
index 58b5aff..9773182 100644
--- a/analysis/webservice/redirect/RedirectHandler.py
+++ b/analysis/webservice/redirect/RedirectHandler.py
@@ -1,4 +1,5 @@
-import tornado
+import tornado.web
+import tornado.gen
import logging
from webservice.webmodel.RequestParameters import RequestParameters
[incubator-sdap-nexus] 04/06: Updated matchup output fields. Validate matchup 'parameter' param against insitu schema
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit 2bab8cb23b23487848e24c310834e8e96be25d98
Author: skorper <st...@gmail.com>
AuthorDate: Mon Aug 22 08:22:58 2022 -0700
Updated matchup output fields. Validate matchup 'parameter' param against insitu schema
---
CHANGELOG.md | 4 ++-
analysis/webservice/algorithms_spark/Matchup.py | 39 ++++++++++++++-----------
2 files changed, 25 insertions(+), 18 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aa47d27..3d97eef 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,13 +14,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- SDAP-371: Renamed `/domssubset` endpoint to `/cdmssubset`
- SDAP-390: Updated NetCDF reader tool for data matchup and added user functionality.
- SDAP-396: Added saildrone insitu api to matchup
+- Matchup validates insitu parameter using insitu API schema endpoint
### Changed
-
-SDAP-390: Changed `/doms` to `/cdms` and `doms_reader.py` to `cdms_reader.py`
- domslist endpoint points to AWS insitu instead of doms insitu
+- Matchup returns numSecondary and numPrimary counts rather than insitu/gridded
### Deprecated
### Removed
- removed dropdown from matchup doms endpoint secondary param
+- Matchup no longer returns insituMatched and griddedMatched fields
### Fixed
- Fix failing test_matchup unit test
- Fixed bug in OpenAPI spec where both matchup endpoints shared the same id
diff --git a/analysis/webservice/algorithms_spark/Matchup.py b/analysis/webservice/algorithms_spark/Matchup.py
index b85c7b4..5dafb31 100644
--- a/analysis/webservice/algorithms_spark/Matchup.py
+++ b/analysis/webservice/algorithms_spark/Matchup.py
@@ -15,7 +15,6 @@
from typing import Optional
-import json
import logging
import threading
from shapely.geometry import Polygon
@@ -30,9 +29,7 @@ import requests
from pytz import timezone, UTC
from scipy import spatial
from shapely import wkt
-from shapely.geometry import Point
from shapely.geometry import box
-from shapely.geos import WKTReadingError
from webservice.NexusHandler import nexus_handler
from webservice.algorithms_spark.NexusCalcSparkHandler import NexusCalcSparkHandler
@@ -46,6 +43,7 @@ from webservice.webmodel import NexusProcessingException
EPOCH = timezone('UTC').localize(datetime(1970, 1, 1))
ISO_8601 = '%Y-%m-%dT%H:%M:%S%z'
+insitu_schema = query_insitu_schema()
def iso_time_to_epoch(str_time):
@@ -73,7 +71,7 @@ class Matchup(NexusCalcSparkHandler):
"parameter": {
"name": "Match-Up Parameter",
"type": "string",
- "description": "The parameter of interest used for the match up. One of 'sst', 'sss', 'wind'. Optional"
+ "description": "The parameter of interest used for the match up. Only used for satellite to insitu matchups. Optional"
},
"startTime": {
"name": "Start Time",
@@ -157,9 +155,10 @@ class Matchup(NexusCalcSparkHandler):
raise NexusProcessingException(reason="'secondary' argument is required", code=400)
parameter_s = request.get_argument('parameter')
- if parameter_s and parameter_s not in ['sst', 'sss', 'wind']:
+ insitu_params = get_insitu_params(insitu_schema)
+ if parameter_s and parameter_s not in insitu_params:
raise NexusProcessingException(
- reason="Parameter %s not supported. Must be one of 'sst', 'sss', 'wind'." % parameter_s, code=400)
+ reason=f"Parameter {parameter_s} not supported. Must be one of {insitu_params}", code=400)
try:
start_time = request.get_start_datetime()
@@ -262,10 +261,8 @@ class Matchup(NexusCalcSparkHandler):
total_values = sum(len(v) for v in spark_result.values())
details = {
"timeToComplete": int((end - start).total_seconds()),
- "numInSituRecords": 0,
- "numInSituMatched": total_values,
- "numGriddedChecked": 0,
- "numGriddedMatched": total_keys
+ "numSecondaryMatched": total_values,
+ "numPrimaryMatched": total_keys
}
matches = Matchup.convert_to_matches(spark_result)
@@ -432,13 +429,8 @@ class DomsPoint(object):
point.device = DomsPoint._variables_to_device(tile.variables)
return point
- insitu_schema = None
-
@staticmethod
def from_edge_point(edge_point):
- if DomsPoint.insitu_schema is None:
- DomsPoint.insitu_schema = query_insitu_schema()
-
point = DomsPoint()
x, y = edge_point['longitude'], edge_point['latitude']
@@ -513,7 +505,7 @@ class DomsPoint(object):
val = edge_point.get(name)
if not val:
continue
- unit = get_insitu_unit(name, DomsPoint.insitu_schema)
+ unit = get_insitu_unit(name, insitu_schema)
data.append(DataPoint(
variable_name=name,
cf_variable_name=name,
@@ -649,9 +641,22 @@ def add_meters_to_lon_lat(lon, lat, meters):
return longitude, latitude
+def get_insitu_params(insitu_schema):
+ """
+ Get all possible insitu params from the CDMS insitu schema
+ """
+ params = insitu_schema.get(
+ 'definitions', {}).get('observation', {}).get('properties', {})
+
+ # Filter params so only variables with units are considered
+ params = list(map(
+ lambda param: param[0], filter(lambda param: 'units'in param[1], params.items())))
+ return params
+
+
def get_insitu_unit(variable_name, insitu_schema):
"""
- Query the insitu API and retrieve the units for the given variable.
+ Retrieve the units from the insitu api schema endpoint for the given variable.
If no units are available for this variable, return "None"
"""
properties = insitu_schema.get('definitions', {}).get('observation', {}).get('properties', {})
[incubator-sdap-nexus] 02/06: Merge branch 'master' of github.com:apache/incubator-sdap-nexus into proxy-bugfix-apidocs
Posted by sk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch proxy-bugfix-apidocs
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
commit 1d5f43d85574dcbe482af10e6768c63ec2b0d006
Merge: 32784eb 1dc62e2
Author: skorper <st...@gmail.com>
AuthorDate: Tue Aug 9 14:11:43 2022 -0700
Merge branch 'master' of github.com:apache/incubator-sdap-nexus into proxy-bugfix-apidocs
CHANGELOG.md | 8 +-
tools/cdms/README.md | 85 +++++++++++++++
tools/cdms/cdms_reader.py | 250 ++++++++++++++++++++++++++++++++++++++++++++
tools/cdms/requirements.txt | 205 ++++++++++++++++++++++++++++++++++++
tools/doms/README.md | 66 ------------
tools/doms/doms_reader.py | 144 -------------------------
6 files changed, 547 insertions(+), 211 deletions(-)