You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sdap.apache.org by sk...@apache.org on 2022/09/02 23:45:15 UTC
[incubator-sdap-nexus] branch master updated: Matchup domsresults bugfix (#197)
This is an automated email from the ASF dual-hosted git repository.
skperez pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
The following commit(s) were added to refs/heads/master by this push:
new 8d497d9 Matchup domsresults bugfix (#197)
8d497d9 is described below
commit 8d497d9a1a96bece2e3c5cf1f28b07e481f068d9
Author: Stepheny Perez <sk...@users.noreply.github.com>
AuthorDate: Fri Sep 2 16:45:10 2022 -0700
Matchup domsresults bugfix (#197)
* Added environment variable to Dockerfile to fix cert error
* Updated openapi spec to link to cdmsschema endpoint
* Fixed matchup db insert. Use new names
* renamed 'details' field for matchup doms to match cdms changes
* Added provider table to openapi spec
* fixed typo
* Updated changelog
---
CHANGELOG.md | 3 +++
.../webservice/algorithms/doms/ResultsStorage.py | 8 ++++----
.../webservice/algorithms_spark/MatchupDoms.py | 6 ++----
analysis/webservice/apidocs/openapi.yml | 24 +++++++++++++++++++---
docker/nexus-webapp/Dockerfile | 3 ++-
5 files changed, 32 insertions(+), 12 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 526a182..84b7a16 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- SDAP-398: Added script for regression tests.
- Matchup validates insitu parameter using insitu API schema endpoint
- Added domsresults endpoint to openapi spec
+- Added markdown table to matchup `platform` param in openapi spec
### Changed
- SDAP-390: Changed `/doms` to `/cdms` and `doms_reader.py` to `cdms_reader.py`
- domslist endpoint points to AWS insitu instead of doms insitu
@@ -40,6 +41,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed CSV and NetCDF matchup output bug
- Fixed NetCDF output switching latitude and longitude
- Fixed import error causing `/timeSeriesSpark` queries to fail.
+- Fixed bug where domsresults no longer worked after successful matchup
+- Fixed certificate error in Dockerfile
### Security
diff --git a/analysis/webservice/algorithms/doms/ResultsStorage.py b/analysis/webservice/algorithms/doms/ResultsStorage.py
index e9005b7..a228737 100644
--- a/analysis/webservice/algorithms/doms/ResultsStorage.py
+++ b/analysis/webservice/algorithms/doms/ResultsStorage.py
@@ -145,10 +145,10 @@ class ResultsStorage(AbstractResultsContainer):
"""
self._session.execute(cql, (
execution_id,
- stats["numGriddedMatched"],
- stats["numGriddedChecked"],
- stats["numInSituMatched"],
- stats["numInSituRecords"],
+ stats["numPrimaryMatched"],
+ None,
+ stats["numSecondaryMatched"],
+ None,
stats["timeToComplete"]
))
diff --git a/analysis/webservice/algorithms_spark/MatchupDoms.py b/analysis/webservice/algorithms_spark/MatchupDoms.py
index 97b3b9e..114b08b 100644
--- a/analysis/webservice/algorithms_spark/MatchupDoms.py
+++ b/analysis/webservice/algorithms_spark/MatchupDoms.py
@@ -266,10 +266,8 @@ class MatchupDoms(NexusCalcSparkHandler):
total_values = sum(len(v) for v in spark_result.values())
details = {
"timeToComplete": int((end - start).total_seconds()),
- "numInSituRecords": 0,
- "numInSituMatched": total_values,
- "numGriddedChecked": 0,
- "numGriddedMatched": total_keys
+ "numSecondaryMatched": total_values,
+ "numPrimaryMatched": total_keys
}
matches = MatchupDoms.convert_to_matches(spark_result)
diff --git a/analysis/webservice/apidocs/openapi.yml b/analysis/webservice/apidocs/openapi.yml
index 18de2f2..38fb358 100644
--- a/analysis/webservice/apidocs/openapi.yml
+++ b/analysis/webservice/apidocs/openapi.yml
@@ -78,7 +78,24 @@ paths:
example: -45,15,-30,30
- in: query
name: platforms
- description: Platforms to include for matchup consideration. Platform depends on which insitu dataset is selected. See `/get_cdms_schema` endpoint for a list of valid platform ids.
+ description: |
+ Platforms to include for matchup consideration. Platform
+ depends on which insitu dataset is selected. Use platform ID
+ in the matchup query.
+
+ <details>
+ <summary>Expand to see a list of valid platforms IDs:</summary>
+ | Platform ID | Platform Name | Web Link |
+ | ----------- | --------------------------------------- | ---------------------------------------------------------- |
+ | 3B | autonomous surface water vehicle | [Link](http://vocab.nerc.ac.uk/collection/L06/current/3B/) |
+ | 0 | unknown | [Link](http://vocab.nerc.ac.uk/collection/L06/current/0/) |
+ | 16 | offshore structure | [Link](http://vocab.nerc.ac.uk/collection/L06/current/16/) |
+ | 17 | coastal structure | [Link](http://vocab.nerc.ac.uk/collection/L06/current/17/) |
+ | 30 | ship | [Link](http://vocab.nerc.ac.uk/collection/L06/current/30/) |
+ | 41 | moored surface buoy | [Link](http://vocab.nerc.ac.uk/collection/L06/current/41/) |
+ | 42 | drifting surface float | [Link](http://vocab.nerc.ac.uk/collection/L06/current/42/) |
+ </details>
+
required: true
schema:
type: string
@@ -124,8 +141,9 @@ paths:
name: parameter
description: |
The parameter of interest used for the match up. This is
- only used for insitu measurements. See `/cdmsschema` endpoint
- for a list of valid options.
+ only used for insitu measurements. See
+ [/get_cdms_schema](https://doms.jpl.nasa.gov/insitu/1.0/cdms_schema)
+ endpoint for a list of valid options.
required: false
schema:
type: string
diff --git a/docker/nexus-webapp/Dockerfile b/docker/nexus-webapp/Dockerfile
index fc58e79..afb3b17 100644
--- a/docker/nexus-webapp/Dockerfile
+++ b/docker/nexus-webapp/Dockerfile
@@ -32,7 +32,8 @@ ENV \
SPARK_HOME=/opt/spark \
PYSPARK_DRIVER_PYTHON=/opt/conda/bin/python3.8 \
PYSPARK_PYTHON=/opt/conda/bin/python3.8 \
- LD_LIBRARY_PATH=/usr/lib
+ LD_LIBRARY_PATH=/usr/lib \
+ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
RUN apk add --update --no-cache \
bzip2 \