You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sdap.apache.org by nc...@apache.org on 2021/10/25 00:08:08 UTC
[incubator-sdap-in-situ-data-services] 01/01: SDAP-354: Import
initial version of in situ data services
This is an automated email from the ASF dual-hosted git repository.
nchung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-in-situ-data-services.git
commit b68a24fbc38f29f5df19d14cea873cc9b1c8810f
Author: nchung <ng...@jpl.nasa.gov>
AuthorDate: Sun Oct 24 17:07:18 2021 -0700
SDAP-354: Import initial version of in situ data services
---
.gitignore | 12 +
CONTRIBUTING.md | 130 ++
LICENSE | 201 +++
README.md | 14 +
flask_server.py | 57 +
in_situ_record_schema.json | 2051 +++++++++++++++++++++
in_situ_schema.json | 2071 ++++++++++++++++++++++
parquet_flask/__init__.py | 27 +
parquet_flask/aws/__init__.py | 14 +
parquet_flask/aws/aws_cred.py | 41 +
parquet_flask/aws/aws_ddb.py | 335 ++++
parquet_flask/aws/aws_s3.py | 117 ++
parquet_flask/io_logic/__init__.py | 14 +
parquet_flask/io_logic/cdms_constants.py | 39 +
parquet_flask/io_logic/ingest_new_file.py | 81 +
parquet_flask/io_logic/metadata_tbl_interface.py | 38 +
parquet_flask/io_logic/metadata_tbl_io.py | 57 +
parquet_flask/io_logic/query.py | 157 ++
parquet_flask/io_logic/query_v2.py | 414 +++++
parquet_flask/io_logic/replace_file.py | 62 +
parquet_flask/io_logic/retrieve_spark_session.py | 62 +
parquet_flask/io_logic/sanitize_record.py | 97 +
parquet_flask/utils/__init__.py | 14 +
parquet_flask/utils/config.py | 46 +
parquet_flask/utils/file_utils.py | 90 +
parquet_flask/utils/general_utils.py | 32 +
parquet_flask/utils/parallel_json_validator.py | 82 +
parquet_flask/utils/singleton.py | 23 +
parquet_flask/utils/time_utils.py | 35 +
parquet_flask/v1/__init__.py | 39 +
parquet_flask/v1/apidocs.py | 32 +
parquet_flask/v1/apidocs/index.html | 61 +
parquet_flask/v1/apidocs/insitu-spec-0.0.1.yml | 263 +++
parquet_flask/v1/ingest_aws_json.py | 156 ++
parquet_flask/v1/ingest_json_s3.py | 58 +
parquet_flask/v1/query_data.py | 89 +
parquet_flask/v1/query_data_doms.py | 132 ++
parquet_flask/v1/replace_json_s3.py | 62 +
setup.py | 52 +
39 files changed, 7357 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..a49984f
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,12 @@
+build
+dist
+*.egg-info
+saildrone*
+local_notes.txt
+ZMFR_sample.json
+scratch*
+.idea
+.run
+CDMS_insitu.parquet
+__pycache__
+.DS_Store
\ No newline at end of file
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..ae9eac8
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,130 @@
+Contributing to Apache SDAP
+===========================
+
+Summary
+-------
+This document covers how to contribute to the SDAP project. SDAP uses github PRs to manage code contributions and project manages source code development through the [SDAP JIRA instance](https://issues.apache.org/jira/browse/SDAP).
+These instructions assume you have a GitHub.com account, so if you don't have one you will have to create one. Your proposed code changes will be published to your own fork of the SDAP project and you will submit a Pull Request for your changes to be added.
+
+_Lets get started!!!_
+
+Bug fixes
+---------
+
+It's very important that we can easily track bug fix commits, so their hashes should remain the same in all branches.
+Therefore, a pull request (PR) that fixes a bug, should be sent against a release branch.
+This can be either the "current release" or the "previous release", depending on which ones are maintained.
+Since the goal is a stable master, bug fixes should be "merged forward" to the next branch in order: "previous release" -> "current release" -> master (in other words: old to new)
+
+Developing new features
+-----------------------
+
+Development should be done in a feature branch, branched off of master.
+Send a PR(steps below) to get it into master (2x LGTM applies).
+PR will only be merged when master is open, will be held otherwise until master is open again.
+No back porting / cherry-picking features to existing branches!
+
+Fork the code
+-------------
+
+In your browser, navigate to: [https://github.com/apache?utf8=✓&q=incubator-sdap&type=&language=](https://github.com/apache?utf8=✓&q=incubator-sdap&type=&language=)
+
+Fork whichever repository you wish to contribute to by clicking on the 'Fork' button on the top right hand side. The fork will happen and you will be taken to your own fork of the repository. Copy the Git repository URL by clicking on the clipboard next to the URL on the right hand side of the page under '**HTTPS** clone URL'. You will paste this URL when doing the following `git clone` command.
+
+On your computer, follow these steps to setup a local repository for working on ACS:
+
+``` bash
+$ git clone https://github.com/YOUR_ACCOUNT/incubator-sdap-*.git
+$ cd incubator-sdap-*
+$ git remote add upstream https://github.com/apache/incubator-sdap-*.git
+$ git checkout master
+$ git fetch upstream
+$ git rebase upstream/master
+```
+N.B. make sure that you replace ```incubator-sdap-*``` with the actual project you wish to contribute to!!!
+
+Making changes
+--------------
+
+It is important that you create a new branch to make changes on and that you do not change the `master` branch (other than to rebase in changes from `upstream/master`). In this example we will assume you will be making your changes to a branch called `SDAP-XXX`. This `SDAP-XXX` is named after the issue you have created within the [SDAP JIRA instance](https://issues.apache.org/jira/browse/SDAP). Therefore `SDAP-XXX` will be created on your local repository and will be pushed to your for [...]
+
+It is best practice to create a new branch each time you want to contribute to the project and only track the changes for that pull request in this branch.
+
+``` bash
+$ git checkout -b SDAP-XXX
+ (make your changes)
+$ git status
+$ git add .
+$ git commit -a -m "SDAP-XXX Descriptive title of SDAP-XXX"
+```
+
+> The `-b` specifies that you want to create a new branch called `SDAP-XXX`. You only specify `-b` the first time you checkout because you are creating a new branch. Once the `SDAP-XXX` branch exists, you can later switch to it with only `git checkout SDAP-XXX`.
+> Note that the commit message comprises the JIRA issue number and title... this makes explicit reference between Github and JIRA for improved project management.
+
+
+Rebase `SDAP-XXX` to include updates from `upstream/master`
+------------------------------------------------------------
+
+It is important that you maintain an up-to-date `master` branch in your local repository. This is done by rebasing in the code changes from `upstream/master` (the official SDAP project repository) into your local repository. You will want to do this before you start working on a feature as well as right before you submit your changes as a pull request. We recommend you do this process periodically while you work to make sure you are working off the most recent project code.
+
+This process will do the following:
+
+1. Checkout your local `master` branch
+2. Synchronize your local `master` branch with the `upstream/master` so you have all the latest changes from the project
+3. Rebase the latest project code into your `SDAP-XXX` branch so it is up-to-date with the upstream code
+
+``` bash
+$ git checkout master
+$ git fetch upstream
+$ git rebase upstream/master
+$ git checkout SDAP-XXX
+$ git rebase master
+```
+
+> Now your `SDAP-XXX` branch is up-to-date with all the code in `upstream/master`.
+
+
+Make a GitHub Pull Request to contribute your changes
+-----------------------------------------------------
+
+When you are happy with your changes and you are ready to contribute them, you will create a Pull Request on GitHub to do so. This is done by pushing your local changes to your forked repository (default remote name is `origin`) and then initiating a pull request on GitHub.
+
+Please include JIRA id, detailed information about the bug/feature, what all tests are executed, how the reviewer can test this feature etc. Incase of UI PRs, a screenshot is preferred.
+
+> **IMPORTANT:** Make sure you have rebased your `SDAP-XXX` branch to include the latest code from `upstream/master` _before_ you do this.
+
+``` bash
+$ git push origin master
+$ git push origin SDAP-XXX
+```
+
+Now that the `SDAP-XXX` branch has been pushed to your GitHub repository, you can initiate the pull request.
+
+To initiate the pull request, do the following:
+
+1. In your browser, navigate to your forked repository: [https://github.com/YOUR_ACCOUNT?utf8=✓&q=incubator-sdap&type=&language=](https://github.com/YOUR_ACCOUNT?utf8=✓&q=incubator-sdap&type=&language=), make sure you actually navigate to the specific project you wish to make the PR from.
+2. Click the new button called '**Compare & pull request**' that showed up just above the main area in your forked repository
+3. Validate the pull request will be into the upstream `master` and will be from your `SDAP-XXX` branch
+4. Enter a detailed description of the work you have done and then click '**Send pull request**'
+
+If you are requested to make modifications to your proposed changes, make the changes locally on your `SDAP-XXX` branch, re-push the `SDAP-XXX` branch to your fork. The existing pull request should automatically pick up the change and update accordingly.
+
+
+Cleaning up after a successful pull request
+-------------------------------------------
+
+Once the `SDAP-XXX` branch has been committed into the `upstream/master` branch, your local `SDAP-XXX` branch and the `origin/SDAP-XXX` branch are no longer needed. If you want to make additional changes, restart the process with a new branch.
+
+> **IMPORTANT:** Make sure that your changes are in `upstream/master` before you delete your `SDAP-XXX` and `origin/SDAP-XXX` branches!
+
+You can delete these deprecated branches with the following:
+
+``` bash
+$ git checkout master
+$ git branch -D SDAP-XXX
+$ git push origin :SDAP-XXX
+```
+
+Release Principles
+------------------
+Coming soon
\ No newline at end of file
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..8dada3e
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "{}"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright {yyyy} {name of copyright owner}
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b573410
--- /dev/null
+++ b/README.md
@@ -0,0 +1,14 @@
+# parquet_test_1
+
+### Ref:
+- how to replace parquet file partially
+```
+https://stackoverflow.com/questions/38487667/overwrite-specific-partitions-in-spark-dataframe-write-method?noredirect=1&lq=1
+> Finally! This is now a feature in Spark 2.3.0: SPARK-20236
+> To use it, you need to set the spark.sql.sources.partitionOverwriteMode setting to dynamic, the dataset needs to be partitioned, and the write mode overwrite. Example:
+> https://stackoverflow.com/questions/50006526/overwrite-only-some-partitions-in-a-partitioned-spark-dataset
+
+spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
+data.toDF().write.mode("overwrite").format("parquet").partitionBy("date", "name").save("s3://path/to/somewhere")
+
+```
\ No newline at end of file
diff --git a/flask_server.py b/flask_server.py
new file mode 100644
index 0000000..688622d
--- /dev/null
+++ b/flask_server.py
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+
+import findspark
+findspark.init()
+
+def flask_me():
+ import logging
+ log_format = '%(asctime)s [%(levelname)s] [%(name)s::%(lineno)d] %(message)s'
+ log_formatter = logging.Formatter(log_format)
+ log_level = getattr(logging, os.getenv('log_level', 'INFO').upper(), None)
+ if not isinstance(log_level, int):
+ print(f'invalid log_level:{log_level}. setting to INFO')
+ log_level = logging.INFO
+
+ file_handler = logging.FileHandler('/tmp/parquet_flask.log', mode='a')
+ file_handler.setLevel(log_level)
+ file_handler.setFormatter(log_formatter)
+
+ stream_handler = logging.StreamHandler(stream=sys.stdout)
+ stream_handler.setLevel(level=log_level)
+ stream_handler.setFormatter(log_formatter)
+
+ logging.basicConfig(
+ level=log_level,
+ format=log_format,
+ handlers=[file_handler, stream_handler]
+ )
+
+ LOGGER = logging.getLogger(__name__)
+ LOGGER.setLevel(logging.INFO)
+
+ from gevent.pywsgi import WSGIServer
+ from parquet_flask import get_app
+ # get_app().run(host='0.0.0.0', port=9788, threaded=True)
+ http_server = WSGIServer(('', 9801), get_app())
+ http_server.serve_forever()
+ return
+
+
+if __name__ == '__main__':
+ flask_me()
diff --git a/in_situ_record_schema.json b/in_situ_record_schema.json
new file mode 100644
index 0000000..2281c48
--- /dev/null
+++ b/in_situ_record_schema.json
@@ -0,0 +1,2051 @@
+{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "title": "Cloud-based Data Match-Up Service In Situ Schema",
+ "description": "Description for each variable is copied from CF standard names table found at https://cfconventions.org/Data/cf-standard-names/77/build/cf-standard-name-table.html",
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "time": {
+ "description": "Time of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0115/",
+ "type": "string",
+ "format": "date-time"
+ },
+ "latitude": {
+ "description": "Latitude position of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0600/",
+ "type": "number",
+ "minimum": -90,
+ "maximum": 90
+ },
+ "longitude": {
+ "description": "Longitude position of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0554/",
+ "type": "number",
+ "minimum": -180,
+ "maximum": 180
+ },
+ "depth": {
+ "description": "Depth value of the observation with below sea level being represented as positive values and -99999 as unknown",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0721/",
+ "type": "number"
+ },
+ "air_pressure": {
+ "description": "Air pressure is the force per unit area which would be exerted when the moving gas molecules of which the air is composed strike a theoretical surface of any orientation.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0015/",
+ "type": "number",
+ "units": "hPa"
+ },
+ "air_pressure_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "air_temperature": {
+ "description": "Air temperature is the bulk temperature of the air, not the surface (skin) temperature.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023/",
+ "type": "number",
+ "units": "C"
+ },
+ "air_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "dew_point_temperature": {
+ "description": "Dew point temperature is the temperature at which a parcel of air reaches saturation upon being cooled at constant pressure and specific humidity.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0723/",
+ "type": "number",
+ "units": "C"
+ },
+ "dew_point_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_longwave_flux_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"longwave\" means longwave radiation. When thought of as being incident on a surface, a radiative flux is sometimes called \"irradiance\". In addition, it is identical with the quantity measured by a cosine-collector light-meter and sometimes called \"vector irradiance\". [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0699/",
+ "type": "number",
+ "units": "W m-2"
+ },
+ "downwelling_longwave_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_longwave_radiance_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"longwave\" means longwave radiation. Radiance is the radiative flux in a particular direction, per unit of solid angle. The direction from which it is coming must be specified, for instance with a coordinate of zenith_angle. If the radiation does not depend on direction, [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0672/",
+ "type": "number",
+ "units": "W m-2 sr-1"
+ },
+ "downwelling_longwave_radiance_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_shortwave_flux_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"shortwave\" means shortwave radiation. When thought of as being incident on a surface, a radiative flux is sometimes called \"irradiance\". In addition, it is identical with the quantity measured by a cosine-collector light-meter and sometimes called \"vector irradiance\" [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0660/",
+ "type": "number",
+ "units": "W m-2"
+ },
+ "downwelling_shortwave_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "mass_concentration_of_chlorophyll_in_sea_water": {
+ "description": "Mass concentration means mass per unit volume and is used in the construction mass_concentration_of_X_in_Y, where X is a material constituent of Y. A chemical species denoted by X may be described by a single term such as \"nitrogen\" or a phrase such as \"nox_expressed_as_nitrogen\". Chlorophylls are the green pigments found in most plants, algae and cyanobacteria; their presence is essential for photosynthesis to take place. There are several different forms of ch [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CF12S21/",
+ "type": "number",
+ "units": "kg m-3"
+ },
+ "mass_concentration_of_chlorophyll_in_sea_water_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "rainfall_rate": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0410/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "rainfall_rate_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "relative_humidity": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0413/",
+ "type": "number",
+ "units": "1"
+ },
+ "relative_humidity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_salinity": {
+ "description": "Sea surface salinity is the salt content of sea water close to the sea surface, often on the Practical Salinity Scale of 1978. However, the unqualified term 'salinity' is generic and does not necessarily imply any particular method of calculation. The units of salinity are dimensionless and the units attribute should normally be given as 1e-3 or 0.001 i.e. parts per thousand. Sea surface salinity is often abbreviated as \"SSS\". For the salinity of sea water at a pa [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0376/",
+ "type": "number",
+ "units": "1e-3"
+ },
+ "sea_surface_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_skin_temperature": {
+ "description": "The sea surface skin temperature is the temperature measured by an infrared radiometer typically operating at wavelengths in the range 3.7 - 12 micrometers. It represents the temperature within the conductive diffusion-dominated sub-layer at a depth of approximately 10 - 20 micrometers below the air-sea interface. Measurements of this quantity are subject to a large potential diurnal cycle including cool skin layer effects (especially at night under clear skies and [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFV9N2/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_skin_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_subskin_temperature": {
+ "description": "The sea surface subskin temperature is the temperature at the base of the conductive laminar sub-layer of the ocean surface, that is, at a depth of approximately 1 - 1.5 millimeters below the air-sea interface. For practical purposes, this quantity can be well approximated to the measurement of surface temperature by a microwave radiometer operating in the 6 - 11 gigahertz frequency range, but the relationship is neither direct nor invariant to changing physical con [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFV9N3/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_subskin_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_temperature": {
+ "description": "Sea surface temperature is usually abbreviated as \"SST\". It is the temperature of sea water near the surface (including the part under sea-ice, if any). More specific terms, namely sea_surface_skin_temperature, sea_surface_subskin_temperature, and surface_temperature are available for the skin, subskin, and interface temperature. respectively. For the temperature of sea water at a particular depth or layer, a data variable of sea_water_temperature with a vertical [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0381/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_density": {
+ "description": "Sea water density is the in-situ density (not the potential density). If 1000 kg m-3 is subtracted, the standard name sea_water_sigma_t should be chosen instead.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0393/",
+ "type": "number",
+ "units": "kg m-3"
+ },
+ "sea_water_density_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_electrical_conductivity": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0394/",
+ "type": "number",
+ "units": "S m-1"
+ },
+ "sea_water_electrical_conductivity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_practical_salinity": {
+ "description": "Practical Salinity, S_P, is a determination of the salinity of sea water, based on its electrical conductance. The measured conductance, corrected for temperature and pressure, is compared to the conductance of a standard potassium chloride solution, producing a value on the Practical Salinity Scale of 1978 (PSS-78). This name should not be used to describe salinity observations made before 1978, or ones not based on conductance measurements. Conversion of Practical [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/IADIHDIJ/",
+ "type": "number",
+ "units": "1"
+ },
+ "sea_water_practical_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_salinity": {
+ "description": "Sea water salinity is the salt content of sea water, often on the Practical Salinity Scale of 1978. However, the unqualified term 'salinity' is generic and does not necessarily imply any particular method of calculation. The units of salinity are dimensionless and the units attribute should normally be given as 1e-3 or 0.001 i.e. parts per thousand. There are standard names for the more precisely defined salinity quantities: sea_water_knudsen_salinity, S_K (used for [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0331/",
+ "type": "number",
+ "units": "1e-3"
+ },
+ "sea_water_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_temperature": {
+ "description": "Sea water temperature is the in situ temperature of the sea water. To specify the depth at which the temperature applies use a vertical coordinate variable or scalar coordinate variable. There are standard names for sea_surface_temperature, sea_surface_skin_temperature, sea_surface_subskin_temperature and sea_surface_foundation_temperature which can be used to describe data located at the specified surfaces. For observed data, depending on the period during which th [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0335/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_water_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "surface_downwelling_photosynthetic_photon_flux_in_air": {
+ "description": "The surface called \"surface\" means the lower boundary of the atmosphere. Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. \"Photosynthetic\" radiation is the part of the spectrum which is used in photosynthesis e.g. 400-700 nm. The range of wavelengths could be specified precisely by the bounds of a coordinate of radiation_wavel [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0284/",
+ "type": "number",
+ "units": "mol m-2 s-1"
+ },
+ "surface_downwelling_photosynthetic_photon_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wet_bulb_temperature": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0035/",
+ "type": "number",
+ "units": "C"
+ },
+ "wet_bulb_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_speed": {
+ "description": "Speed is the magnitude of velocity. Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) The wind speed is the magnitude of the wind velocity.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0038/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "wind_speed_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_from_direction": {
+ "description": "Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) In meteorological reports, the direction of the wind vector is usually (but not always) given as the direction from which it is blowing (wind_from_direction) (westerly, northerly, etc.). In other contexts, such as atmospheric modelling, it is often natural to give the direction in the usual [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0036/",
+ "type": "number",
+ "units": "degree"
+ },
+ "wind_to_direction": {
+ "description": "Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) In meteorological reports, the direction of the wind vector is usually (but not always) given as the direction from which it is blowing (wind_from_direction) (westerly, northerly, etc.). In other contexts, such as atmospheric modelling, it is often natural to give the direction in the usual [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0041/",
+ "type": "number",
+ "units": "degree"
+ },
+ "wind_from_direction_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_to_direction_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "eastward_wind": {
+ "description": "\"Eastward\" indicates a vector component which is positive when directed eastward (negative westward). Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.)",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0653/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "northward_wind": {
+ "description": "\"Northward\" indicates a vector component which is positive when directed northward (negative southward). Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.)",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0461/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "wind_component_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "platform": {
+ "description": "",
+ "$ref": "#/definitions/platform"
+ },
+ "device": {
+ "description": "Device identifier is provided by the Natural Environment Research Council (NERC) Vocabulary Server developed and maintained by the British Oceanographic Data Centre and can be found at http://vocab.nerc.ac.uk/collection/L05/current/",
+ "type": "string",
+ "oneOf": [
+ {
+ "const": "364",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/364/",
+ "title": "1000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "363",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/363/",
+ "title": "1000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "365",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/365/",
+ "title": "2000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "372",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/372/",
+ "title": "2000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "375",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/375/",
+ "title": "250 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "362",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/362/",
+ "title": "250 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "374",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/374/",
+ "title": "60 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "371",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/371/",
+ "title": "60 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "366",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/366/",
+ "title": ">2000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "373",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/373/",
+ "title": ">2000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "367",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/367/",
+ "title": ">2000 Hz top-bandwidth sub-bottom penetrator and mud profiler systems"
+ },
+ {
+ "const": "384",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/384/",
+ "title": "ADVs and turbulence probes"
+ },
+ {
+ "const": "386",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/386/",
+ "title": "Aerosol physical characterisers"
+ },
+ {
+ "const": "387",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/387/",
+ "title": "Aethalometers"
+ },
+ {
+ "const": "AQPL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/AQPL/",
+ "title": "Aquapulse"
+ },
+ {
+ "const": "POS16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS16/",
+ "title": "Argos GPS-localised transmitters"
+ },
+ {
+ "const": "POS17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS17/",
+ "title": "Argos doppler-localised transmitters"
+ },
+ {
+ "const": "MOD03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD03/",
+ "title": "Atmosphere models"
+ },
+ {
+ "const": "314",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/314/",
+ "title": "Atmospheric chemistry remote sensors"
+ },
+ {
+ "const": "LAB49",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB49/",
+ "title": "Atomic fluorescence spectrometers"
+ },
+ {
+ "const": "POS10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS10/",
+ "title": "Automatic Ranging Grid Overlay DM 54 receivers"
+ },
+ {
+ "const": "MOD08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD08/",
+ "title": "Biological and biogeochemical models"
+ },
+ {
+ "const": "BOMR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/BOMR/",
+ "title": "Boomer"
+ },
+ {
+ "const": "130",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/130/",
+ "title": "CTD"
+ },
+ {
+ "const": "131",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/131/",
+ "title": "CTD undulators"
+ },
+ {
+ "const": "CHRP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CHRP/",
+ "title": "Chirp"
+ },
+ {
+ "const": "COLCMP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/COLCMP/",
+ "title": "Colour comparators"
+ },
+ {
+ "const": "MOD05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD05/",
+ "title": "Coupled models"
+ },
+ {
+ "const": "POS08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS08/",
+ "title": "Decca Navigator System HiFix receivers"
+ },
+ {
+ "const": "POS01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS01/",
+ "title": "Decca Navigator System main chain receivers"
+ },
+ {
+ "const": "POS04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS04/",
+ "title": "Differential Global Positioning System receivers"
+ },
+ {
+ "const": "389",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/389/",
+ "title": "Expendable bathythermographs"
+ },
+ {
+ "const": "FFES",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FFES/",
+ "title": "Fish-finder echosounders"
+ },
+ {
+ "const": "FLXS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FLXS/",
+ "title": "Flexichoc"
+ },
+ {
+ "const": "FLXT",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FLXT/",
+ "title": "Flexotir"
+ },
+ {
+ "const": "GIAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/GIAG/",
+ "title": "GI-gun"
+ },
+ {
+ "const": "LAB39",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB39/",
+ "title": "Geiger counters"
+ },
+ {
+ "const": "POS02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS02/",
+ "title": "Global Navigation Satellite System receivers"
+ },
+ {
+ "const": "MOD01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD01/",
+ "title": "Global models"
+ },
+ {
+ "const": "POS21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS21/",
+ "title": "Hyperfix receivers"
+ },
+ {
+ "const": "POS25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS25/",
+ "title": "Kinematic Global Positioning System receivers"
+ },
+ {
+ "const": "LVLST",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LVLST/",
+ "title": "Levels and staffs"
+ },
+ {
+ "const": "POS05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS05/",
+ "title": "Long Range Navigation version C receivers"
+ },
+ {
+ "const": "POS06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS06/",
+ "title": "Marine chronometers"
+ },
+ {
+ "const": "390",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/390/",
+ "title": "Mechanical bathythermographs"
+ },
+ {
+ "const": "MOD06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD06/",
+ "title": "Meteorological models"
+ },
+ {
+ "const": "POS20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS20/",
+ "title": "Motorola Mini-Ranger receivers"
+ },
+ {
+ "const": "POS03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS03/",
+ "title": "NAVSTAR Global Positioning System receivers"
+ },
+ {
+ "const": "POS18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS18/",
+ "title": "Navy Navigation Satellite System receivers"
+ },
+ {
+ "const": "MOD04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD04/",
+ "title": "Ocean models"
+ },
+ {
+ "const": "POS15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS15/",
+ "title": "Omega receivers"
+ },
+ {
+ "const": "PERMON",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/PERMON/",
+ "title": "Performance monitors"
+ },
+ {
+ "const": "MOD07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD07/",
+ "title": "Physical oceanographic models"
+ },
+ {
+ "const": "394",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/394/",
+ "title": "Pinger sub-bottom profilers"
+ },
+ {
+ "const": "POS14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS14/",
+ "title": "Pulse8 receivers"
+ },
+ {
+ "const": "POS13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS13/",
+ "title": "Rana receivers"
+ },
+ {
+ "const": "MOD02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD02/",
+ "title": "Regional models"
+ },
+ {
+ "const": "POS07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS07/",
+ "title": "Sextants"
+ },
+ {
+ "const": "SPRK",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SPRK/",
+ "title": "Sparker"
+ },
+ {
+ "const": "POS12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS12/",
+ "title": "Syst\ufffdme leger de measure de distance receivers"
+ },
+ {
+ "const": "POS09",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS09/",
+ "title": "Toran receivers"
+ },
+ {
+ "const": "POS11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS11/",
+ "title": "Trident receivers"
+ },
+ {
+ "const": "POS22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS22/",
+ "title": "Trisponder navigation systems"
+ },
+ {
+ "const": "POS26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS26/",
+ "title": "Ultra Short Baseline Positioning Systems"
+ },
+ {
+ "const": "VSHC",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/VSHC/",
+ "title": "Vaporchoc"
+ },
+ {
+ "const": "LAB40",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB40/",
+ "title": "Volume measures"
+ },
+ {
+ "const": "WDRG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/WDRG/",
+ "title": "Wire drags"
+ },
+ {
+ "const": "LAB26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB26/",
+ "title": "X-ray diffractometers"
+ },
+ {
+ "const": "LAB25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB25/",
+ "title": "X-ray fluorescence analysers"
+ },
+ {
+ "const": "LAB17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB17/",
+ "title": "accelerator mass spectrometers"
+ },
+ {
+ "const": "307",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/307/",
+ "title": "accelerometers"
+ },
+ {
+ "const": "183",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/183/",
+ "title": "acoustic backscatter sensors"
+ },
+ {
+ "const": "357",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/357/",
+ "title": "acoustic tracking systems"
+ },
+ {
+ "const": "LAB32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB32/",
+ "title": "acoustic velocity systems"
+ },
+ {
+ "const": "64",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/64/",
+ "title": "active fishing gear"
+ },
+ {
+ "const": "353",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/353/",
+ "title": "active fluorometers"
+ },
+ {
+ "const": "13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/13/",
+ "title": "aerosol samplers"
+ },
+ {
+ "const": "SNAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SNAG/",
+ "title": "airgun"
+ },
+ {
+ "const": "ARAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ARAG/",
+ "title": "airgun array"
+ },
+ {
+ "const": "379",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/379/",
+ "title": "altimeters"
+ },
+ {
+ "const": "101",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/101/",
+ "title": "anemometers"
+ },
+ {
+ "const": "382",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/382/",
+ "title": "atmospheric gas analysers"
+ },
+ {
+ "const": "308",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/308/",
+ "title": "atmospheric radiometers"
+ },
+ {
+ "const": "ATRANS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ATRANS/",
+ "title": "atmospheric transparency quantifiers"
+ },
+ {
+ "const": "LAB10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB10/",
+ "title": "atomic absorption spectrometers"
+ },
+ {
+ "const": "LAB04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB04/",
+ "title": "autoanalysers"
+ },
+ {
+ "const": "LAB13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB13/",
+ "title": "balances and scales"
+ },
+ {
+ "const": "312",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/312/",
+ "title": "bathymetric LiDARs"
+ },
+ {
+ "const": "132",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/132/",
+ "title": "bathythermographs"
+ },
+ {
+ "const": "62",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/62/",
+ "title": "beam trawls"
+ },
+ {
+ "const": "LAB23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB23/",
+ "title": "bench fluorometers"
+ },
+ {
+ "const": "LAB27",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB27/",
+ "title": "bench particle sizers"
+ },
+ {
+ "const": "151",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/151/",
+ "title": "benthic incubation chambers"
+ },
+ {
+ "const": "24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/24/",
+ "title": "benthos samplers"
+ },
+ {
+ "const": "393",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/393/",
+ "title": "binoculars and terrestrial scopes"
+ },
+ {
+ "const": "BLUMS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/BLUMS/",
+ "title": "bioluminescence sensors"
+ },
+ {
+ "const": "85",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/85/",
+ "title": "biorectors and chemostats"
+ },
+ {
+ "const": "313",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/313/",
+ "title": "bird detection radars"
+ },
+ {
+ "const": "358",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/358/",
+ "title": "bubble detectors"
+ },
+ {
+ "const": "311",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/311/",
+ "title": "cameras"
+ },
+ {
+ "const": "LAB38",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB38/",
+ "title": "cavity enhanced absorption spectrometers"
+ },
+ {
+ "const": "83",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/83/",
+ "title": "centrifuges"
+ },
+ {
+ "const": "LAB34",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB34/",
+ "title": "chemiluminescence analysers"
+ },
+ {
+ "const": "CCOV",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CCOV/",
+ "title": "cloud cover quantifiers"
+ },
+ {
+ "const": "392",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/392/",
+ "title": "colonisation substrata"
+ },
+ {
+ "const": "LAB03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB03/",
+ "title": "colorimeters"
+ },
+ {
+ "const": "58",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/58/",
+ "title": "continuous air samplers"
+ },
+ {
+ "const": "31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/31/",
+ "title": "continuous water samplers"
+ },
+ {
+ "const": "CSEM",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CSEM/",
+ "title": "controlled-source electromagnetic systems"
+ },
+ {
+ "const": "114",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/114/",
+ "title": "current meters"
+ },
+ {
+ "const": "115",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/115/",
+ "title": "current profilers"
+ },
+ {
+ "const": "DLOG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/DLOG/",
+ "title": "data loggers"
+ },
+ {
+ "const": "POS24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS24/",
+ "title": "dead reckoning"
+ },
+ {
+ "const": "61",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/61/",
+ "title": "demersal trawl nets"
+ },
+ {
+ "const": "11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/11/",
+ "title": "discrete air samplers"
+ },
+ {
+ "const": "30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/30/",
+ "title": "discrete water samplers"
+ },
+ {
+ "const": "351",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/351/",
+ "title": "dissolved gas sensors"
+ },
+ {
+ "const": "ERTQ",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ERTQ/",
+ "title": "earthquake"
+ },
+ {
+ "const": "EMSR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EMSR/",
+ "title": "electromagnetic sensors"
+ },
+ {
+ "const": "LAB08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB08/",
+ "title": "electron microprobes"
+ },
+ {
+ "const": "LAB07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB07/",
+ "title": "electron microscopes"
+ },
+ {
+ "const": "LAB01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB01/",
+ "title": "elemental analysers"
+ },
+ {
+ "const": "EQUIL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EQUIL/",
+ "title": "equilibrators"
+ },
+ {
+ "const": "354",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/354/",
+ "title": "expendable CTDs"
+ },
+ {
+ "const": "70",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/70/",
+ "title": "fish aggregating devices"
+ },
+ {
+ "const": "LAB37",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB37/",
+ "title": "flow cytometers"
+ },
+ {
+ "const": "LAB36",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB36/",
+ "title": "flow injection analysers"
+ },
+ {
+ "const": "388",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/388/",
+ "title": "flow meters"
+ },
+ {
+ "const": "LAB06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB06/",
+ "title": "fluorescence microscopes"
+ },
+ {
+ "const": "113",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/113/",
+ "title": "fluorometers"
+ },
+ {
+ "const": "66",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/66/",
+ "title": "flushing drills"
+ },
+ {
+ "const": "12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/12/",
+ "title": "fog samplers"
+ },
+ {
+ "const": "LAB22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB22/",
+ "title": "gamma-ray spectrometers"
+ },
+ {
+ "const": "LAB19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB19/",
+ "title": "gas chromatograph mass spectrometers"
+ },
+ {
+ "const": "LAB02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB02/",
+ "title": "gas chromatographs"
+ },
+ {
+ "const": "LAB24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB24/",
+ "title": "gel permeation chromatographs"
+ },
+ {
+ "const": "370",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/370/",
+ "title": "geothermometers"
+ },
+ {
+ "const": "65",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/65/",
+ "title": "gill nets"
+ },
+ {
+ "const": "158",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/158/",
+ "title": "gravimeters"
+ },
+ {
+ "const": "EXLH",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EXLH/",
+ "title": "high penetration low resolution explosives"
+ },
+ {
+ "const": "LAB11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB11/",
+ "title": "high performance liquid chromatographs"
+ },
+ {
+ "const": "EXHL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EXHL/",
+ "title": "high resolution low penetration explosives"
+ },
+ {
+ "const": "67",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/67/",
+ "title": "high-speed plankton samplers"
+ },
+ {
+ "const": "LAB51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB51/",
+ "title": "holographic microscopes"
+ },
+ {
+ "const": "369",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/369/",
+ "title": "hydrophones"
+ },
+ {
+ "const": "ICEP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ICEP/",
+ "title": "ice thickness profilers"
+ },
+ {
+ "const": "82",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/82/",
+ "title": "in-situ incubators"
+ },
+ {
+ "const": "150",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/150/",
+ "title": "in-situ particle sizers"
+ },
+ {
+ "const": "182",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/182/",
+ "title": "inapplicable"
+ },
+ {
+ "const": "LAB14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB14/",
+ "title": "inductively-coupled plasma atomic emission spectroscopes"
+ },
+ {
+ "const": "LAB15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB15/",
+ "title": "inductively-coupled plasma mass spectrometers"
+ },
+ {
+ "const": "POS19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS19/",
+ "title": "inertial navigation systems"
+ },
+ {
+ "const": "86",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/86/",
+ "title": "inorganic carbon analysers"
+ },
+ {
+ "const": "112",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/112/",
+ "title": "inverted echosounders"
+ },
+ {
+ "const": "LAB45",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB45/",
+ "title": "ion chromatography analysers"
+ },
+ {
+ "const": "LAB09",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB09/",
+ "title": "ion microprobes"
+ },
+ {
+ "const": "LAB48",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB48/",
+ "title": "isotope ratio mass spectrometers"
+ },
+ {
+ "const": "LBTG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LBTG/",
+ "title": "labelling tags"
+ },
+ {
+ "const": "LAB33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB33/",
+ "title": "laboratory autosamplers"
+ },
+ {
+ "const": "LAB42",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB42/",
+ "title": "laboratory optical rangefinders"
+ },
+ {
+ "const": "310",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/310/",
+ "title": "laser altimeters"
+ },
+ {
+ "const": "LAB21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB21/",
+ "title": "liquid scintillation counters"
+ },
+ {
+ "const": "116",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/116/",
+ "title": "lowered current profilers"
+ },
+ {
+ "const": "LAB31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB31/",
+ "title": "magnetic susceptibility systems"
+ },
+ {
+ "const": "159",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/159/",
+ "title": "magnetometers"
+ },
+ {
+ "const": "90",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/90/",
+ "title": "manual biota samplers"
+ },
+ {
+ "const": "57",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/57/",
+ "title": "manual rock samplers"
+ },
+ {
+ "const": "MVBR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MVBR/",
+ "title": "marine vibroseis"
+ },
+ {
+ "const": "LAB16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB16/",
+ "title": "mass spectrometers"
+ },
+ {
+ "const": "383",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/383/",
+ "title": "metal analysers"
+ },
+ {
+ "const": "92",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/92/",
+ "title": "meteorological LiDARs"
+ },
+ {
+ "const": "102",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/102/",
+ "title": "meteorological packages"
+ },
+ {
+ "const": "184",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/184/",
+ "title": "microstructure sensors"
+ },
+ {
+ "const": "157",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/157/",
+ "title": "multi-beam echosounders"
+ },
+ {
+ "const": "154",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/154/",
+ "title": "multi-channel seismic reflection systems"
+ },
+ {
+ "const": "MMIS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MMIS/",
+ "title": "multi-frame imaging sonar"
+ },
+ {
+ "const": "68",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/68/",
+ "title": "multinet"
+ },
+ {
+ "const": "NATS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/NATS/",
+ "title": "natural source"
+ },
+ {
+ "const": "69",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/69/",
+ "title": "neuston net"
+ },
+ {
+ "const": "LAB18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB18/",
+ "title": "nuclear magnetic resonance spectrometers"
+ },
+ {
+ "const": "181",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/181/",
+ "title": "nutrient analysers"
+ },
+ {
+ "const": "91",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/91/",
+ "title": "observers"
+ },
+ {
+ "const": "304",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/304/",
+ "title": "ocean colour radiometers"
+ },
+ {
+ "const": "123",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/123/",
+ "title": "optical backscatter sensors"
+ },
+ {
+ "const": "LAB05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB05/",
+ "title": "optical microscopes"
+ },
+ {
+ "const": "355",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/355/",
+ "title": "pH sensors"
+ },
+ {
+ "const": "32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/32/",
+ "title": "particulates samplers"
+ },
+ {
+ "const": "63",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/63/",
+ "title": "passive fishing gear"
+ },
+ {
+ "const": "23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/23/",
+ "title": "pelagic trawl nets"
+ },
+ {
+ "const": "361",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/361/",
+ "title": "penetrometers"
+ },
+ {
+ "const": "360",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/360/",
+ "title": "piezometers"
+ },
+ {
+ "const": "352",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/352/",
+ "title": "plankton categorisers and counters"
+ },
+ {
+ "const": "22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/22/",
+ "title": "plankton nets"
+ },
+ {
+ "const": "21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/21/",
+ "title": "plankton recorders"
+ },
+ {
+ "const": "LAB43",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB43/",
+ "title": "plate readers"
+ },
+ {
+ "const": "385",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/385/",
+ "title": "platform attitude sensors"
+ },
+ {
+ "const": "359",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/359/",
+ "title": "plummets"
+ },
+ {
+ "const": "381",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/381/",
+ "title": "precipitation gauges"
+ },
+ {
+ "const": "14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/14/",
+ "title": "precipitation samplers"
+ },
+ {
+ "const": "LAB46",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB46/",
+ "title": "proton transfer reaction mass spectrometers"
+ },
+ {
+ "const": "186",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/186/",
+ "title": "radar altimeters"
+ },
+ {
+ "const": "POS23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS23/",
+ "title": "radio tracking systems"
+ },
+ {
+ "const": "122",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/122/",
+ "title": "radiometers"
+ },
+ {
+ "const": "103",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/103/",
+ "title": "radiosondes"
+ },
+ {
+ "const": "356",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/356/",
+ "title": "redox potential sensors"
+ },
+ {
+ "const": "LAB44",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB44/",
+ "title": "refractometers"
+ },
+ {
+ "const": "53",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/53/",
+ "title": "rock corers"
+ },
+ {
+ "const": "54",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/54/",
+ "title": "rock dredges"
+ },
+ {
+ "const": "LAB29",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB29/",
+ "title": "rulers"
+ },
+ {
+ "const": "350",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/350/",
+ "title": "salinity sensor"
+ },
+ {
+ "const": "LAB30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB30/",
+ "title": "salinometers"
+ },
+ {
+ "const": "59",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/59/",
+ "title": "sampling extent markers"
+ },
+ {
+ "const": "301",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/301/",
+ "title": "satellite positioning systems"
+ },
+ {
+ "const": "121",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/121/",
+ "title": "satellite tracking system"
+ },
+ {
+ "const": "305",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/305/",
+ "title": "scatterometers"
+ },
+ {
+ "const": "111",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/111/",
+ "title": "sea level recorders"
+ },
+ {
+ "const": "60",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/60/",
+ "title": "sediment dredges"
+ },
+ {
+ "const": "50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/50/",
+ "title": "sediment grabs"
+ },
+ {
+ "const": "52",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/52/",
+ "title": "sediment porewater samplers"
+ },
+ {
+ "const": "378",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/378/",
+ "title": "sediment profile imagers"
+ },
+ {
+ "const": "55",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/55/",
+ "title": "sediment settling tubes"
+ },
+ {
+ "const": "391",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/391/",
+ "title": "sediment suction samplers"
+ },
+ {
+ "const": "376",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/376/",
+ "title": "sediment surface markers"
+ },
+ {
+ "const": "33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/33/",
+ "title": "sediment traps"
+ },
+ {
+ "const": "155",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/155/",
+ "title": "seismic refraction systems"
+ },
+ {
+ "const": "368",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/368/",
+ "title": "seismometers"
+ },
+ {
+ "const": "81",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/81/",
+ "title": "shipboard incubators"
+ },
+ {
+ "const": "152",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/152/",
+ "title": "sidescan sonars"
+ },
+ {
+ "const": "84",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/84/",
+ "title": "sieves and filters"
+ },
+ {
+ "const": "156",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/156/",
+ "title": "single-beam echosounders"
+ },
+ {
+ "const": "SBAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SBAG/",
+ "title": "single-bubble airgun"
+ },
+ {
+ "const": "153",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/153/",
+ "title": "single-channel seismic reflection systems"
+ },
+ {
+ "const": "56",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/56/",
+ "title": "snow and ice samplers"
+ },
+ {
+ "const": "185",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/185/",
+ "title": "sound velocity sensors"
+ },
+ {
+ "const": "LAB20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB20/",
+ "title": "spectrophotometers"
+ },
+ {
+ "const": "380",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/380/",
+ "title": "submarine cables"
+ },
+ {
+ "const": "303",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/303/",
+ "title": "surface current radars"
+ },
+ {
+ "const": "306",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/306/",
+ "title": "synthetic aperture radars"
+ },
+ {
+ "const": "309",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/309/",
+ "title": "terrestrial radiometers"
+ },
+ {
+ "const": "LAB28",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB28/",
+ "title": "thermal conductivimeters"
+ },
+ {
+ "const": "LAB50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB50/",
+ "title": "thermal cyclers"
+ },
+ {
+ "const": "LAB47",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB47/",
+ "title": "thermal ionisation mass spectrometers"
+ },
+ {
+ "const": "135",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/135/",
+ "title": "thermistor chains"
+ },
+ {
+ "const": "133",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/133/",
+ "title": "thermosalinographs"
+ },
+ {
+ "const": "TFSAMP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/TFSAMP/",
+ "title": "thin film metal samplers"
+ },
+ {
+ "const": "LAB12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB12/",
+ "title": "titrators"
+ },
+ {
+ "const": "TRTG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/TRTG/",
+ "title": "tracking tags"
+ },
+ {
+ "const": "124",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/124/",
+ "title": "transmissometers"
+ },
+ {
+ "const": "51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/51/",
+ "title": "unconsolidated sediment corers"
+ },
+ {
+ "const": "999",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/999/",
+ "title": "unknown"
+ },
+ {
+ "const": "LAB35",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB35/",
+ "title": "voltammetry analysers"
+ },
+ {
+ "const": "302",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/302/",
+ "title": "water body temperature sensor"
+ },
+ {
+ "const": "377",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/377/",
+ "title": "water level markers"
+ },
+ {
+ "const": "WPS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/WPS/",
+ "title": "water pressure sensors"
+ },
+ {
+ "const": "134",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/134/",
+ "title": "water temperature sensor"
+ },
+ {
+ "const": "SNWG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SNWG/",
+ "title": "watergun"
+ },
+ {
+ "const": "110",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/110/",
+ "title": "wave recorders"
+ }
+ ]
+ },
+ "meta": {
+ "description": "Data provider specific metadata pertaining to the observation",
+ "type": [
+ "string",
+ "object"
+ ]
+ }
+ },
+ "required": [
+ "time",
+ "latitude",
+ "longitude",
+ "depth"
+ ],
+ "minProperties": 5,
+ "definitions": {
+ "platform": {
+ "description": "",
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "code": {
+ "description": "Platform identifier is provided by the Natural Environment Research Council (NERC) Vocabulary Server developed and maintained by the British Oceanographic Data Centre and can be found at http://vocab.nerc.ac.uk/collection/L06/current/",
+ "type": "string",
+ "oneOf": [
+ {
+ "const": "9A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/9A/",
+ "title": "DUKW"
+ },
+ {
+ "const": "4A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/4A/",
+ "title": "Ice-tethered subsurface profiling float"
+ },
+ {
+ "const": "62",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/62/",
+ "title": "aeroplane"
+ },
+ {
+ "const": "54",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/54/",
+ "title": "airship"
+ },
+ {
+ "const": "96",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/96/",
+ "title": "amphibious crawler"
+ },
+ {
+ "const": "95",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/95/",
+ "title": "amphibious vehicle"
+ },
+ {
+ "const": "69",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/69/",
+ "title": "autogyro"
+ },
+ {
+ "const": "3B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3B/",
+ "title": "autonomous surface water vehicle"
+ },
+ {
+ "const": "25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/25/",
+ "title": "autonomous underwater vehicle"
+ },
+ {
+ "const": "13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/13/",
+ "title": "beach/intertidal zone structure"
+ },
+ {
+ "const": "50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/50/",
+ "title": "buoyant aircraft"
+ },
+ {
+ "const": "75",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/75/",
+ "title": "cetacean"
+ },
+ {
+ "const": "17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/17/",
+ "title": "coastal structure"
+ },
+ {
+ "const": "90",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/90/",
+ "title": "cryosphere"
+ },
+ {
+ "const": "72",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/72/",
+ "title": "diver"
+ },
+ {
+ "const": "94",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/94/",
+ "title": "drift ice"
+ },
+ {
+ "const": "24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/24/",
+ "title": "drifting manned submersible"
+ },
+ {
+ "const": "44",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/44/",
+ "title": "drifting subsurface float"
+ },
+ {
+ "const": "46",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/46/",
+ "title": "drifting subsurface profiling float"
+ },
+ {
+ "const": "42",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/42/",
+ "title": "drifting surface float"
+ },
+ {
+ "const": "3D",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3D/",
+ "title": "drillship"
+ },
+ {
+ "const": "76",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/76/",
+ "title": "fish"
+ },
+ {
+ "const": "36",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/36/",
+ "title": "fishing vessel"
+ },
+ {
+ "const": "11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/11/",
+ "title": "fixed benthic node"
+ },
+ {
+ "const": "45",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/45/",
+ "title": "fixed subsurface vertical profiler"
+ },
+ {
+ "const": "73",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/73/",
+ "title": "flightless bird"
+ },
+ {
+ "const": "47",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/47/",
+ "title": "float"
+ },
+ {
+ "const": "52",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/52/",
+ "title": "free-floating balloon"
+ },
+ {
+ "const": "51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/51/",
+ "title": "free-rising balloon"
+ },
+ {
+ "const": "64",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/64/",
+ "title": "geostationary orbiting satellite"
+ },
+ {
+ "const": "6A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6A/",
+ "title": "glider"
+ },
+ {
+ "const": "67",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/67/",
+ "title": "helicopter"
+ },
+ {
+ "const": "9B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/9B/",
+ "title": "hovercraft"
+ },
+ {
+ "const": "71",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/71/",
+ "title": "human"
+ },
+ {
+ "const": "91",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/91/",
+ "title": "ice island"
+ },
+ {
+ "const": "92",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/92/",
+ "title": "ice shelf"
+ },
+ {
+ "const": "6B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6B/",
+ "title": "kite"
+ },
+ {
+ "const": "10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/10/",
+ "title": "land or seafloor"
+ },
+ {
+ "const": "77",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/77/",
+ "title": "land-sea mammals"
+ },
+ {
+ "const": "14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/14/",
+ "title": "land/onshore structure"
+ },
+ {
+ "const": "15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/15/",
+ "title": "land/onshore vehicle"
+ },
+ {
+ "const": "26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/26/",
+ "title": "lowered unmanned submersible"
+ },
+ {
+ "const": "38",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/38/",
+ "title": "man-powered boat"
+ },
+ {
+ "const": "3A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3A/",
+ "title": "man-powered small boat"
+ },
+ {
+ "const": "66",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/66/",
+ "title": "manned spacecraft"
+ },
+ {
+ "const": "19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/19/",
+ "title": "mesocosm bag"
+ },
+ {
+ "const": "41",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/41/",
+ "title": "moored surface buoy"
+ },
+ {
+ "const": "48",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/48/",
+ "title": "mooring"
+ },
+ {
+ "const": "39",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/39/",
+ "title": "naval vessel"
+ },
+ {
+ "const": "60",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/60/",
+ "title": "non-buoyant aircraft"
+ },
+ {
+ "const": "16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/16/",
+ "title": "offshore structure"
+ },
+ {
+ "const": "65",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/65/",
+ "title": "orbiting satellite"
+ },
+ {
+ "const": "70",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/70/",
+ "title": "organism"
+ },
+ {
+ "const": "93",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/93/",
+ "title": "pack ice"
+ },
+ {
+ "const": "6C",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6C/",
+ "title": "parachute"
+ },
+ {
+ "const": "21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/21/",
+ "title": "propelled manned submersible"
+ },
+ {
+ "const": "22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/22/",
+ "title": "propelled unmanned submersible"
+ },
+ {
+ "const": "61",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/61/",
+ "title": "research aeroplane"
+ },
+ {
+ "const": "31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/31/",
+ "title": "research vessel"
+ },
+ {
+ "const": "18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/18/",
+ "title": "river station"
+ },
+ {
+ "const": "63",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/63/",
+ "title": "rocket"
+ },
+ {
+ "const": "68",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/68/",
+ "title": "satellite"
+ },
+ {
+ "const": "12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/12/",
+ "title": "sea bed vehicle"
+ },
+ {
+ "const": "74",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/74/",
+ "title": "seabird and duck"
+ },
+ {
+ "const": "37",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/37/",
+ "title": "self-propelled boat"
+ },
+ {
+ "const": "33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/33/",
+ "title": "self-propelled small boat"
+ },
+ {
+ "const": "30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/30/",
+ "title": "ship"
+ },
+ {
+ "const": "6Z",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6Z/",
+ "title": "spacecraft"
+ },
+ {
+ "const": "27",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/27/",
+ "title": "sub-surface gliders"
+ },
+ {
+ "const": "20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/20/",
+ "title": "submersible"
+ },
+ {
+ "const": "43",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/43/",
+ "title": "subsurface mooring"
+ },
+ {
+ "const": "3C",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3C/",
+ "title": "surface gliders"
+ },
+ {
+ "const": "49",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/49/",
+ "title": "surface ice buoy"
+ },
+ {
+ "const": "3Z",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3Z/",
+ "title": "surface vessel"
+ },
+ {
+ "const": "53",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/53/",
+ "title": "tethered balloon"
+ },
+ {
+ "const": "23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/23/",
+ "title": "towed unmanned submersible"
+ },
+ {
+ "const": "0",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/0/",
+ "title": "unknown"
+ },
+ {
+ "const": "6D",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6D/",
+ "title": "unmanned aerial vehicle"
+ },
+ {
+ "const": "34",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/34/",
+ "title": "vessel at fixed position"
+ },
+ {
+ "const": "32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/32/",
+ "title": "vessel of opportunity"
+ },
+ {
+ "const": "35",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/35/",
+ "title": "vessel of opportunity on fixed route"
+ }
+ ]
+ },
+ "id": {
+ "description": "Optional platform identifier",
+ "type": "string"
+ },
+ "type": {
+ "description": "Optional platform type",
+ "type": "string"
+ }
+ },
+ "required": [
+ "code"
+ ]
+ },
+ "quality": {
+ "description": "International Oceanographic Data and Information Exchange (IODE) primary level quality flag scheme. Full guide can be found at https://www.nodc.noaa.gov/oads/support/MG54_3.pdf",
+ "type": "integer",
+ "oneOf": [
+ {
+ "const": 1,
+ "title": "Good",
+ "description": "Passed documented required QC tests"
+ },
+ {
+ "const": 2,
+ "title": "Not evaluated, not available or unknown",
+ "description": "Used for data when no QC test performed or the information on quality is not available"
+ },
+ {
+ "const": 3,
+ "title": "Questionable/suspect",
+ "description": "Failed non-critical documented metric or subjective test(s)"
+ },
+ {
+ "const": 4,
+ "title": "Bad",
+ "description": "Failed critical documented QC test(s) or as assigned by the data provider"
+ },
+ {
+ "const": 9,
+ "title": "Missing data",
+ "description": "Used as place holder when data are missing"
+ }
+ ]
+ }
+ }
+}
\ No newline at end of file
diff --git a/in_situ_schema.json b/in_situ_schema.json
new file mode 100644
index 0000000..9e0482d
--- /dev/null
+++ b/in_situ_schema.json
@@ -0,0 +1,2071 @@
+{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "title": "Cloud-based Data Match-Up Service In Situ Schema",
+ "description": "Schema for in situ data",
+ "properties": {
+ "provider": {
+ "description": "",
+ "type": "string"
+ },
+ "project": {
+ "description": "",
+ "type": "string"
+ },
+ "observations": {
+ "type": "array",
+ "items": {
+ "$ref": "#/definitions/observation"
+ },
+ "minItems": 1
+ }
+ },
+ "definitions": {
+ "platform": {
+ "description": "",
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "code": {
+ "description": "Platform identifier is provided by the Natural Environment Research Council (NERC) Vocabulary Server developed and maintained by the British Oceanographic Data Centre and can be found at http://vocab.nerc.ac.uk/collection/L06/current/",
+ "type": "string",
+ "oneOf": [
+ {
+ "const": "9A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/9A/",
+ "title": "DUKW"
+ },
+ {
+ "const": "4A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/4A/",
+ "title": "Ice-tethered subsurface profiling float"
+ },
+ {
+ "const": "62",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/62/",
+ "title": "aeroplane"
+ },
+ {
+ "const": "54",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/54/",
+ "title": "airship"
+ },
+ {
+ "const": "96",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/96/",
+ "title": "amphibious crawler"
+ },
+ {
+ "const": "95",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/95/",
+ "title": "amphibious vehicle"
+ },
+ {
+ "const": "69",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/69/",
+ "title": "autogyro"
+ },
+ {
+ "const": "3B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3B/",
+ "title": "autonomous surface water vehicle"
+ },
+ {
+ "const": "25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/25/",
+ "title": "autonomous underwater vehicle"
+ },
+ {
+ "const": "13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/13/",
+ "title": "beach/intertidal zone structure"
+ },
+ {
+ "const": "50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/50/",
+ "title": "buoyant aircraft"
+ },
+ {
+ "const": "75",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/75/",
+ "title": "cetacean"
+ },
+ {
+ "const": "17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/17/",
+ "title": "coastal structure"
+ },
+ {
+ "const": "90",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/90/",
+ "title": "cryosphere"
+ },
+ {
+ "const": "72",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/72/",
+ "title": "diver"
+ },
+ {
+ "const": "94",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/94/",
+ "title": "drift ice"
+ },
+ {
+ "const": "24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/24/",
+ "title": "drifting manned submersible"
+ },
+ {
+ "const": "44",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/44/",
+ "title": "drifting subsurface float"
+ },
+ {
+ "const": "46",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/46/",
+ "title": "drifting subsurface profiling float"
+ },
+ {
+ "const": "42",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/42/",
+ "title": "drifting surface float"
+ },
+ {
+ "const": "3D",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3D/",
+ "title": "drillship"
+ },
+ {
+ "const": "76",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/76/",
+ "title": "fish"
+ },
+ {
+ "const": "36",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/36/",
+ "title": "fishing vessel"
+ },
+ {
+ "const": "11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/11/",
+ "title": "fixed benthic node"
+ },
+ {
+ "const": "45",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/45/",
+ "title": "fixed subsurface vertical profiler"
+ },
+ {
+ "const": "73",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/73/",
+ "title": "flightless bird"
+ },
+ {
+ "const": "47",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/47/",
+ "title": "float"
+ },
+ {
+ "const": "52",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/52/",
+ "title": "free-floating balloon"
+ },
+ {
+ "const": "51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/51/",
+ "title": "free-rising balloon"
+ },
+ {
+ "const": "64",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/64/",
+ "title": "geostationary orbiting satellite"
+ },
+ {
+ "const": "6A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6A/",
+ "title": "glider"
+ },
+ {
+ "const": "67",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/67/",
+ "title": "helicopter"
+ },
+ {
+ "const": "9B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/9B/",
+ "title": "hovercraft"
+ },
+ {
+ "const": "71",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/71/",
+ "title": "human"
+ },
+ {
+ "const": "91",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/91/",
+ "title": "ice island"
+ },
+ {
+ "const": "92",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/92/",
+ "title": "ice shelf"
+ },
+ {
+ "const": "6B",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6B/",
+ "title": "kite"
+ },
+ {
+ "const": "10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/10/",
+ "title": "land or seafloor"
+ },
+ {
+ "const": "77",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/77/",
+ "title": "land-sea mammals"
+ },
+ {
+ "const": "14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/14/",
+ "title": "land/onshore structure"
+ },
+ {
+ "const": "15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/15/",
+ "title": "land/onshore vehicle"
+ },
+ {
+ "const": "26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/26/",
+ "title": "lowered unmanned submersible"
+ },
+ {
+ "const": "38",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/38/",
+ "title": "man-powered boat"
+ },
+ {
+ "const": "3A",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3A/",
+ "title": "man-powered small boat"
+ },
+ {
+ "const": "66",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/66/",
+ "title": "manned spacecraft"
+ },
+ {
+ "const": "19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/19/",
+ "title": "mesocosm bag"
+ },
+ {
+ "const": "41",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/41/",
+ "title": "moored surface buoy"
+ },
+ {
+ "const": "48",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/48/",
+ "title": "mooring"
+ },
+ {
+ "const": "39",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/39/",
+ "title": "naval vessel"
+ },
+ {
+ "const": "60",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/60/",
+ "title": "non-buoyant aircraft"
+ },
+ {
+ "const": "16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/16/",
+ "title": "offshore structure"
+ },
+ {
+ "const": "65",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/65/",
+ "title": "orbiting satellite"
+ },
+ {
+ "const": "70",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/70/",
+ "title": "organism"
+ },
+ {
+ "const": "93",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/93/",
+ "title": "pack ice"
+ },
+ {
+ "const": "6C",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6C/",
+ "title": "parachute"
+ },
+ {
+ "const": "21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/21/",
+ "title": "propelled manned submersible"
+ },
+ {
+ "const": "22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/22/",
+ "title": "propelled unmanned submersible"
+ },
+ {
+ "const": "61",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/61/",
+ "title": "research aeroplane"
+ },
+ {
+ "const": "31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/31/",
+ "title": "research vessel"
+ },
+ {
+ "const": "18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/18/",
+ "title": "river station"
+ },
+ {
+ "const": "63",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/63/",
+ "title": "rocket"
+ },
+ {
+ "const": "68",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/68/",
+ "title": "satellite"
+ },
+ {
+ "const": "12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/12/",
+ "title": "sea bed vehicle"
+ },
+ {
+ "const": "74",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/74/",
+ "title": "seabird and duck"
+ },
+ {
+ "const": "37",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/37/",
+ "title": "self-propelled boat"
+ },
+ {
+ "const": "33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/33/",
+ "title": "self-propelled small boat"
+ },
+ {
+ "const": "30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/30/",
+ "title": "ship"
+ },
+ {
+ "const": "6Z",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6Z/",
+ "title": "spacecraft"
+ },
+ {
+ "const": "27",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/27/",
+ "title": "sub-surface gliders"
+ },
+ {
+ "const": "20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/20/",
+ "title": "submersible"
+ },
+ {
+ "const": "43",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/43/",
+ "title": "subsurface mooring"
+ },
+ {
+ "const": "3C",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3C/",
+ "title": "surface gliders"
+ },
+ {
+ "const": "49",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/49/",
+ "title": "surface ice buoy"
+ },
+ {
+ "const": "3Z",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/3Z/",
+ "title": "surface vessel"
+ },
+ {
+ "const": "53",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/53/",
+ "title": "tethered balloon"
+ },
+ {
+ "const": "23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/23/",
+ "title": "towed unmanned submersible"
+ },
+ {
+ "const": "0",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/0/",
+ "title": "unknown"
+ },
+ {
+ "const": "6D",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/6D/",
+ "title": "unmanned aerial vehicle"
+ },
+ {
+ "const": "34",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/34/",
+ "title": "vessel at fixed position"
+ },
+ {
+ "const": "32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/32/",
+ "title": "vessel of opportunity"
+ },
+ {
+ "const": "35",
+ "uri": "http://vocab.nerc.ac.uk/collection/L06/current/35/",
+ "title": "vessel of opportunity on fixed route"
+ }
+ ]
+ },
+ "id": {
+ "description": "Optional platform identifier",
+ "type": "string"
+ },
+ "type": {
+ "description": "Optional platform type",
+ "type": "string"
+ }
+ },
+ "required": [
+ "code"
+ ]
+ },
+ "quality": {
+ "description": "International Oceanographic Data and Information Exchange (IODE) primary level quality flag scheme. Full guide can be found at https://www.nodc.noaa.gov/oads/support/MG54_3.pdf",
+ "type": "integer",
+ "oneOf": [
+ {
+ "const": 1,
+ "title": "Good",
+ "description": "Passed documented required QC tests"
+ },
+ {
+ "const": 2,
+ "title": "Not evaluated, not available or unknown",
+ "description": "Used for data when no QC test performed or the information on quality is not available"
+ },
+ {
+ "const": 3,
+ "title": "Questionable/suspect",
+ "description": "Failed non-critical documented metric or subjective test(s)"
+ },
+ {
+ "const": 4,
+ "title": "Bad",
+ "description": "Failed critical documented QC test(s) or as assigned by the data provider"
+ },
+ {
+ "const": 9,
+ "title": "Missing data",
+ "description": "Used as place holder when data are missing"
+ }
+ ]
+ },
+ "observation": {
+ "description": "Description for each variable is copied from CF standard names table found at https://cfconventions.org/Data/cf-standard-names/77/build/cf-standard-name-table.html",
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "time": {
+ "description": "Time of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0115/",
+ "type": "string",
+ "format": "date-time"
+ },
+ "latitude": {
+ "description": "Latitude position of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0600/",
+ "type": "number",
+ "minimum": -90,
+ "maximum": 90
+ },
+ "longitude": {
+ "description": "Longitude position of the observation",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0554/",
+ "type": "number",
+ "minimum": -180,
+ "maximum": 180
+ },
+ "depth": {
+ "description": "Depth value of the observation with below sea level being represented as positive values and -99999 as unknown",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0721/",
+ "type": "number"
+ },
+ "air_pressure": {
+ "description": "Air pressure is the force per unit area which would be exerted when the moving gas molecules of which the air is composed strike a theoretical surface of any orientation.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0015/",
+ "type": "number",
+ "units": "hPa"
+ },
+ "air_pressure_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "air_temperature": {
+ "description": "Air temperature is the bulk temperature of the air, not the surface (skin) temperature.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023/",
+ "type": "number",
+ "units": "C"
+ },
+ "air_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "dew_point_temperature": {
+ "description": "Dew point temperature is the temperature at which a parcel of air reaches saturation upon being cooled at constant pressure and specific humidity.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0723/",
+ "type": "number",
+ "units": "C"
+ },
+ "dew_point_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_longwave_flux_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"longwave\" means longwave radiation. When thought of as being incident on a surface, a radiative flux is sometimes called \"irradiance\". In addition, it is identical with the quantity measured by a cosine-collector light-meter and sometimes called \"vector irradiance [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0699/",
+ "type": "number",
+ "units": "W m-2"
+ },
+ "downwelling_longwave_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_longwave_radiance_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"longwave\" means longwave radiation. Radiance is the radiative flux in a particular direction, per unit of solid angle. The direction from which it is coming must be specified, for instance with a coordinate of zenith_angle. If the radiation does not depend on directi [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0672/",
+ "type": "number",
+ "units": "W m-2 sr-1"
+ },
+ "downwelling_longwave_radiance_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "downwelling_shortwave_flux_in_air": {
+ "description": "Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. The term \"shortwave\" means shortwave radiation. When thought of as being incident on a surface, a radiative flux is sometimes called \"irradiance\". In addition, it is identical with the quantity measured by a cosine-collector light-meter and sometimes called \"vector irradian [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0660/",
+ "type": "number",
+ "units": "W m-2"
+ },
+ "downwelling_shortwave_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "mass_concentration_of_chlorophyll_in_sea_water": {
+ "description": "Mass concentration means mass per unit volume and is used in the construction mass_concentration_of_X_in_Y, where X is a material constituent of Y. A chemical species denoted by X may be described by a single term such as \"nitrogen\" or a phrase such as \"nox_expressed_as_nitrogen\". Chlorophylls are the green pigments found in most plants, algae and cyanobacteria; their presence is essential for photosynthesis to take place. There are several different forms o [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CF12S21/",
+ "type": "number",
+ "units": "kg m-3"
+ },
+ "mass_concentration_of_chlorophyll_in_sea_water_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "rainfall_rate": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0410/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "rainfall_rate_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "relative_humidity": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0413/",
+ "type": "number",
+ "units": "1"
+ },
+ "relative_humidity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_salinity": {
+ "description": "Sea surface salinity is the salt content of sea water close to the sea surface, often on the Practical Salinity Scale of 1978. However, the unqualified term 'salinity' is generic and does not necessarily imply any particular method of calculation. The units of salinity are dimensionless and the units attribute should normally be given as 1e-3 or 0.001 i.e. parts per thousand. Sea surface salinity is often abbreviated as \"SSS\". For the salinity of sea water at [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0376/",
+ "type": "number",
+ "units": "1e-3"
+ },
+ "sea_surface_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_skin_temperature": {
+ "description": "The sea surface skin temperature is the temperature measured by an infrared radiometer typically operating at wavelengths in the range 3.7 - 12 micrometers. It represents the temperature within the conductive diffusion-dominated sub-layer at a depth of approximately 10 - 20 micrometers below the air-sea interface. Measurements of this quantity are subject to a large potential diurnal cycle including cool skin layer effects (especially at night under clear skies [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFV9N2/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_skin_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_subskin_temperature": {
+ "description": "The sea surface subskin temperature is the temperature at the base of the conductive laminar sub-layer of the ocean surface, that is, at a depth of approximately 1 - 1.5 millimeters below the air-sea interface. For practical purposes, this quantity can be well approximated to the measurement of surface temperature by a microwave radiometer operating in the 6 - 11 gigahertz frequency range, but the relationship is neither direct nor invariant to changing physical [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFV9N3/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_subskin_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_surface_temperature": {
+ "description": "Sea surface temperature is usually abbreviated as \"SST\". It is the temperature of sea water near the surface (including the part under sea-ice, if any). More specific terms, namely sea_surface_skin_temperature, sea_surface_subskin_temperature, and surface_temperature are available for the skin, subskin, and interface temperature. respectively. For the temperature of sea water at a particular depth or layer, a data variable of sea_water_temperature with a verti [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0381/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_surface_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_density": {
+ "description": "Sea water density is the in-situ density (not the potential density). If 1000 kg m-3 is subtracted, the standard name sea_water_sigma_t should be chosen instead.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0393/",
+ "type": "number",
+ "units": "kg m-3"
+ },
+ "sea_water_density_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_electrical_conductivity": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0394/",
+ "type": "number",
+ "units": "S m-1"
+ },
+ "sea_water_electrical_conductivity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_practical_salinity": {
+ "description": "Practical Salinity, S_P, is a determination of the salinity of sea water, based on its electrical conductance. The measured conductance, corrected for temperature and pressure, is compared to the conductance of a standard potassium chloride solution, producing a value on the Practical Salinity Scale of 1978 (PSS-78). This name should not be used to describe salinity observations made before 1978, or ones not based on conductance measurements. Conversion of Pract [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/IADIHDIJ/",
+ "type": "number",
+ "units": "1"
+ },
+ "sea_water_practical_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_salinity": {
+ "description": "Sea water salinity is the salt content of sea water, often on the Practical Salinity Scale of 1978. However, the unqualified term 'salinity' is generic and does not necessarily imply any particular method of calculation. The units of salinity are dimensionless and the units attribute should normally be given as 1e-3 or 0.001 i.e. parts per thousand. There are standard names for the more precisely defined salinity quantities: sea_water_knudsen_salinity, S_K (used [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0331/",
+ "type": "number",
+ "units": "1e-3"
+ },
+ "sea_water_salinity_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "sea_water_temperature": {
+ "description": "Sea water temperature is the in situ temperature of the sea water. To specify the depth at which the temperature applies use a vertical coordinate variable or scalar coordinate variable. There are standard names for sea_surface_temperature, sea_surface_skin_temperature, sea_surface_subskin_temperature and sea_surface_foundation_temperature which can be used to describe data located at the specified surfaces. For observed data, depending on the period during whic [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0335/",
+ "type": "number",
+ "units": "C"
+ },
+ "sea_water_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "surface_downwelling_photosynthetic_photon_flux_in_air": {
+ "description": "The surface called \"surface\" means the lower boundary of the atmosphere. Downwelling radiation is radiation from above. It does not mean \"net downward\". The sign convention is that \"upwelling\" is positive upwards and \"downwelling\" is positive downwards. \"Photosynthetic\" radiation is the part of the spectrum which is used in photosynthesis e.g. 400-700 nm. The range of wavelengths could be specified precisely by the bounds of a coordinate of radiation_w [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0284/",
+ "type": "number",
+ "units": "mol m-2 s-1"
+ },
+ "surface_downwelling_photosynthetic_photon_flux_in_air_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wet_bulb_temperature": {
+ "description": "",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0035/",
+ "type": "number",
+ "units": "C"
+ },
+ "wet_bulb_temperature_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_speed": {
+ "description": "Speed is the magnitude of velocity. Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) The wind speed is the magnitude of the wind velocity.",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0038/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "wind_speed_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_from_direction": {
+ "description": "Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) In meteorological reports, the direction of the wind vector is usually (but not always) given as the direction from which it is blowing (wind_from_direction) (westerly, northerly, etc.). In other contexts, such as atmospheric modelling, it is often natural to give the direction in the us [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0036/",
+ "type": "number",
+ "units": "degree"
+ },
+ "wind_to_direction": {
+ "description": "Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.) In meteorological reports, the direction of the wind vector is usually (but not always) given as the direction from which it is blowing (wind_from_direction) (westerly, northerly, etc.). In other contexts, such as atmospheric modelling, it is often natural to give the direction in the us [...]
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0041/",
+ "type": "number",
+ "units": "degree"
+ },
+ "wind_from_direction_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "wind_to_direction_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "eastward_wind": {
+ "description": "\"Eastward\" indicates a vector component which is positive when directed eastward (negative westward). Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.)",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0653/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "northward_wind": {
+ "description": "\"Northward\" indicates a vector component which is positive when directed northward (negative southward). Wind is defined as a two-dimensional (horizontal) air velocity vector, with no vertical component. (Vertical motion in the atmosphere has the standard name upward_air_velocity.)",
+ "uri": "http://vocab.nerc.ac.uk/collection/P07/current/CFSN0461/",
+ "type": "number",
+ "units": "m s-1"
+ },
+ "wind_component_quality": {
+ "description": "",
+ "$ref": "#/definitions/quality"
+ },
+ "platform": {
+ "description": "",
+ "$ref": "#/definitions/platform"
+ },
+ "device": {
+ "description": "Device identifier is provided by the Natural Environment Research Council (NERC) Vocabulary Server developed and maintained by the British Oceanographic Data Centre and can be found at http://vocab.nerc.ac.uk/collection/L05/current/",
+ "type": "string",
+ "oneOf": [
+ {
+ "const": "364",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/364/",
+ "title": "1000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "363",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/363/",
+ "title": "1000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "365",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/365/",
+ "title": "2000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "372",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/372/",
+ "title": "2000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "375",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/375/",
+ "title": "250 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "362",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/362/",
+ "title": "250 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "374",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/374/",
+ "title": "60 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "371",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/371/",
+ "title": "60 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "366",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/366/",
+ "title": ">2000 Hz top-bandwidth multi-channel seismic reflection systems"
+ },
+ {
+ "const": "373",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/373/",
+ "title": ">2000 Hz top-bandwidth single-channel seismic reflection systems"
+ },
+ {
+ "const": "367",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/367/",
+ "title": ">2000 Hz top-bandwidth sub-bottom penetrator and mud profiler systems"
+ },
+ {
+ "const": "384",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/384/",
+ "title": "ADVs and turbulence probes"
+ },
+ {
+ "const": "386",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/386/",
+ "title": "Aerosol physical characterisers"
+ },
+ {
+ "const": "387",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/387/",
+ "title": "Aethalometers"
+ },
+ {
+ "const": "AQPL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/AQPL/",
+ "title": "Aquapulse"
+ },
+ {
+ "const": "POS16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS16/",
+ "title": "Argos GPS-localised transmitters"
+ },
+ {
+ "const": "POS17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS17/",
+ "title": "Argos doppler-localised transmitters"
+ },
+ {
+ "const": "MOD03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD03/",
+ "title": "Atmosphere models"
+ },
+ {
+ "const": "314",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/314/",
+ "title": "Atmospheric chemistry remote sensors"
+ },
+ {
+ "const": "LAB49",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB49/",
+ "title": "Atomic fluorescence spectrometers"
+ },
+ {
+ "const": "POS10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS10/",
+ "title": "Automatic Ranging Grid Overlay DM 54 receivers"
+ },
+ {
+ "const": "MOD08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD08/",
+ "title": "Biological and biogeochemical models"
+ },
+ {
+ "const": "BOMR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/BOMR/",
+ "title": "Boomer"
+ },
+ {
+ "const": "130",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/130/",
+ "title": "CTD"
+ },
+ {
+ "const": "131",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/131/",
+ "title": "CTD undulators"
+ },
+ {
+ "const": "CHRP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CHRP/",
+ "title": "Chirp"
+ },
+ {
+ "const": "COLCMP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/COLCMP/",
+ "title": "Colour comparators"
+ },
+ {
+ "const": "MOD05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD05/",
+ "title": "Coupled models"
+ },
+ {
+ "const": "POS08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS08/",
+ "title": "Decca Navigator System HiFix receivers"
+ },
+ {
+ "const": "POS01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS01/",
+ "title": "Decca Navigator System main chain receivers"
+ },
+ {
+ "const": "POS04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS04/",
+ "title": "Differential Global Positioning System receivers"
+ },
+ {
+ "const": "389",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/389/",
+ "title": "Expendable bathythermographs"
+ },
+ {
+ "const": "FFES",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FFES/",
+ "title": "Fish-finder echosounders"
+ },
+ {
+ "const": "FLXS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FLXS/",
+ "title": "Flexichoc"
+ },
+ {
+ "const": "FLXT",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/FLXT/",
+ "title": "Flexotir"
+ },
+ {
+ "const": "GIAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/GIAG/",
+ "title": "GI-gun"
+ },
+ {
+ "const": "LAB39",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB39/",
+ "title": "Geiger counters"
+ },
+ {
+ "const": "POS02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS02/",
+ "title": "Global Navigation Satellite System receivers"
+ },
+ {
+ "const": "MOD01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD01/",
+ "title": "Global models"
+ },
+ {
+ "const": "POS21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS21/",
+ "title": "Hyperfix receivers"
+ },
+ {
+ "const": "POS25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS25/",
+ "title": "Kinematic Global Positioning System receivers"
+ },
+ {
+ "const": "LVLST",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LVLST/",
+ "title": "Levels and staffs"
+ },
+ {
+ "const": "POS05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS05/",
+ "title": "Long Range Navigation version C receivers"
+ },
+ {
+ "const": "POS06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS06/",
+ "title": "Marine chronometers"
+ },
+ {
+ "const": "390",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/390/",
+ "title": "Mechanical bathythermographs"
+ },
+ {
+ "const": "MOD06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD06/",
+ "title": "Meteorological models"
+ },
+ {
+ "const": "POS20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS20/",
+ "title": "Motorola Mini-Ranger receivers"
+ },
+ {
+ "const": "POS03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS03/",
+ "title": "NAVSTAR Global Positioning System receivers"
+ },
+ {
+ "const": "POS18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS18/",
+ "title": "Navy Navigation Satellite System receivers"
+ },
+ {
+ "const": "MOD04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD04/",
+ "title": "Ocean models"
+ },
+ {
+ "const": "POS15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS15/",
+ "title": "Omega receivers"
+ },
+ {
+ "const": "PERMON",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/PERMON/",
+ "title": "Performance monitors"
+ },
+ {
+ "const": "MOD07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD07/",
+ "title": "Physical oceanographic models"
+ },
+ {
+ "const": "394",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/394/",
+ "title": "Pinger sub-bottom profilers"
+ },
+ {
+ "const": "POS14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS14/",
+ "title": "Pulse8 receivers"
+ },
+ {
+ "const": "POS13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS13/",
+ "title": "Rana receivers"
+ },
+ {
+ "const": "MOD02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MOD02/",
+ "title": "Regional models"
+ },
+ {
+ "const": "POS07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS07/",
+ "title": "Sextants"
+ },
+ {
+ "const": "SPRK",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SPRK/",
+ "title": "Sparker"
+ },
+ {
+ "const": "POS12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS12/",
+ "title": "Syst\ufffdme leger de measure de distance receivers"
+ },
+ {
+ "const": "POS09",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS09/",
+ "title": "Toran receivers"
+ },
+ {
+ "const": "POS11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS11/",
+ "title": "Trident receivers"
+ },
+ {
+ "const": "POS22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS22/",
+ "title": "Trisponder navigation systems"
+ },
+ {
+ "const": "POS26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS26/",
+ "title": "Ultra Short Baseline Positioning Systems"
+ },
+ {
+ "const": "VSHC",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/VSHC/",
+ "title": "Vaporchoc"
+ },
+ {
+ "const": "LAB40",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB40/",
+ "title": "Volume measures"
+ },
+ {
+ "const": "WDRG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/WDRG/",
+ "title": "Wire drags"
+ },
+ {
+ "const": "LAB26",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB26/",
+ "title": "X-ray diffractometers"
+ },
+ {
+ "const": "LAB25",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB25/",
+ "title": "X-ray fluorescence analysers"
+ },
+ {
+ "const": "LAB17",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB17/",
+ "title": "accelerator mass spectrometers"
+ },
+ {
+ "const": "307",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/307/",
+ "title": "accelerometers"
+ },
+ {
+ "const": "183",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/183/",
+ "title": "acoustic backscatter sensors"
+ },
+ {
+ "const": "357",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/357/",
+ "title": "acoustic tracking systems"
+ },
+ {
+ "const": "LAB32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB32/",
+ "title": "acoustic velocity systems"
+ },
+ {
+ "const": "64",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/64/",
+ "title": "active fishing gear"
+ },
+ {
+ "const": "353",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/353/",
+ "title": "active fluorometers"
+ },
+ {
+ "const": "13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/13/",
+ "title": "aerosol samplers"
+ },
+ {
+ "const": "SNAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SNAG/",
+ "title": "airgun"
+ },
+ {
+ "const": "ARAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ARAG/",
+ "title": "airgun array"
+ },
+ {
+ "const": "379",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/379/",
+ "title": "altimeters"
+ },
+ {
+ "const": "101",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/101/",
+ "title": "anemometers"
+ },
+ {
+ "const": "382",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/382/",
+ "title": "atmospheric gas analysers"
+ },
+ {
+ "const": "308",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/308/",
+ "title": "atmospheric radiometers"
+ },
+ {
+ "const": "ATRANS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ATRANS/",
+ "title": "atmospheric transparency quantifiers"
+ },
+ {
+ "const": "LAB10",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB10/",
+ "title": "atomic absorption spectrometers"
+ },
+ {
+ "const": "LAB04",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB04/",
+ "title": "autoanalysers"
+ },
+ {
+ "const": "LAB13",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB13/",
+ "title": "balances and scales"
+ },
+ {
+ "const": "312",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/312/",
+ "title": "bathymetric LiDARs"
+ },
+ {
+ "const": "132",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/132/",
+ "title": "bathythermographs"
+ },
+ {
+ "const": "62",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/62/",
+ "title": "beam trawls"
+ },
+ {
+ "const": "LAB23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB23/",
+ "title": "bench fluorometers"
+ },
+ {
+ "const": "LAB27",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB27/",
+ "title": "bench particle sizers"
+ },
+ {
+ "const": "151",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/151/",
+ "title": "benthic incubation chambers"
+ },
+ {
+ "const": "24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/24/",
+ "title": "benthos samplers"
+ },
+ {
+ "const": "393",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/393/",
+ "title": "binoculars and terrestrial scopes"
+ },
+ {
+ "const": "BLUMS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/BLUMS/",
+ "title": "bioluminescence sensors"
+ },
+ {
+ "const": "85",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/85/",
+ "title": "biorectors and chemostats"
+ },
+ {
+ "const": "313",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/313/",
+ "title": "bird detection radars"
+ },
+ {
+ "const": "358",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/358/",
+ "title": "bubble detectors"
+ },
+ {
+ "const": "311",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/311/",
+ "title": "cameras"
+ },
+ {
+ "const": "LAB38",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB38/",
+ "title": "cavity enhanced absorption spectrometers"
+ },
+ {
+ "const": "83",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/83/",
+ "title": "centrifuges"
+ },
+ {
+ "const": "LAB34",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB34/",
+ "title": "chemiluminescence analysers"
+ },
+ {
+ "const": "CCOV",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CCOV/",
+ "title": "cloud cover quantifiers"
+ },
+ {
+ "const": "392",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/392/",
+ "title": "colonisation substrata"
+ },
+ {
+ "const": "LAB03",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB03/",
+ "title": "colorimeters"
+ },
+ {
+ "const": "58",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/58/",
+ "title": "continuous air samplers"
+ },
+ {
+ "const": "31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/31/",
+ "title": "continuous water samplers"
+ },
+ {
+ "const": "CSEM",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/CSEM/",
+ "title": "controlled-source electromagnetic systems"
+ },
+ {
+ "const": "114",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/114/",
+ "title": "current meters"
+ },
+ {
+ "const": "115",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/115/",
+ "title": "current profilers"
+ },
+ {
+ "const": "DLOG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/DLOG/",
+ "title": "data loggers"
+ },
+ {
+ "const": "POS24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS24/",
+ "title": "dead reckoning"
+ },
+ {
+ "const": "61",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/61/",
+ "title": "demersal trawl nets"
+ },
+ {
+ "const": "11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/11/",
+ "title": "discrete air samplers"
+ },
+ {
+ "const": "30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/30/",
+ "title": "discrete water samplers"
+ },
+ {
+ "const": "351",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/351/",
+ "title": "dissolved gas sensors"
+ },
+ {
+ "const": "ERTQ",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ERTQ/",
+ "title": "earthquake"
+ },
+ {
+ "const": "EMSR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EMSR/",
+ "title": "electromagnetic sensors"
+ },
+ {
+ "const": "LAB08",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB08/",
+ "title": "electron microprobes"
+ },
+ {
+ "const": "LAB07",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB07/",
+ "title": "electron microscopes"
+ },
+ {
+ "const": "LAB01",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB01/",
+ "title": "elemental analysers"
+ },
+ {
+ "const": "EQUIL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EQUIL/",
+ "title": "equilibrators"
+ },
+ {
+ "const": "354",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/354/",
+ "title": "expendable CTDs"
+ },
+ {
+ "const": "70",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/70/",
+ "title": "fish aggregating devices"
+ },
+ {
+ "const": "LAB37",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB37/",
+ "title": "flow cytometers"
+ },
+ {
+ "const": "LAB36",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB36/",
+ "title": "flow injection analysers"
+ },
+ {
+ "const": "388",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/388/",
+ "title": "flow meters"
+ },
+ {
+ "const": "LAB06",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB06/",
+ "title": "fluorescence microscopes"
+ },
+ {
+ "const": "113",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/113/",
+ "title": "fluorometers"
+ },
+ {
+ "const": "66",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/66/",
+ "title": "flushing drills"
+ },
+ {
+ "const": "12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/12/",
+ "title": "fog samplers"
+ },
+ {
+ "const": "LAB22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB22/",
+ "title": "gamma-ray spectrometers"
+ },
+ {
+ "const": "LAB19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB19/",
+ "title": "gas chromatograph mass spectrometers"
+ },
+ {
+ "const": "LAB02",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB02/",
+ "title": "gas chromatographs"
+ },
+ {
+ "const": "LAB24",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB24/",
+ "title": "gel permeation chromatographs"
+ },
+ {
+ "const": "370",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/370/",
+ "title": "geothermometers"
+ },
+ {
+ "const": "65",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/65/",
+ "title": "gill nets"
+ },
+ {
+ "const": "158",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/158/",
+ "title": "gravimeters"
+ },
+ {
+ "const": "EXLH",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EXLH/",
+ "title": "high penetration low resolution explosives"
+ },
+ {
+ "const": "LAB11",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB11/",
+ "title": "high performance liquid chromatographs"
+ },
+ {
+ "const": "EXHL",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/EXHL/",
+ "title": "high resolution low penetration explosives"
+ },
+ {
+ "const": "67",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/67/",
+ "title": "high-speed plankton samplers"
+ },
+ {
+ "const": "LAB51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB51/",
+ "title": "holographic microscopes"
+ },
+ {
+ "const": "369",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/369/",
+ "title": "hydrophones"
+ },
+ {
+ "const": "ICEP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/ICEP/",
+ "title": "ice thickness profilers"
+ },
+ {
+ "const": "82",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/82/",
+ "title": "in-situ incubators"
+ },
+ {
+ "const": "150",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/150/",
+ "title": "in-situ particle sizers"
+ },
+ {
+ "const": "182",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/182/",
+ "title": "inapplicable"
+ },
+ {
+ "const": "LAB14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB14/",
+ "title": "inductively-coupled plasma atomic emission spectroscopes"
+ },
+ {
+ "const": "LAB15",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB15/",
+ "title": "inductively-coupled plasma mass spectrometers"
+ },
+ {
+ "const": "POS19",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS19/",
+ "title": "inertial navigation systems"
+ },
+ {
+ "const": "86",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/86/",
+ "title": "inorganic carbon analysers"
+ },
+ {
+ "const": "112",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/112/",
+ "title": "inverted echosounders"
+ },
+ {
+ "const": "LAB45",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB45/",
+ "title": "ion chromatography analysers"
+ },
+ {
+ "const": "LAB09",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB09/",
+ "title": "ion microprobes"
+ },
+ {
+ "const": "LAB48",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB48/",
+ "title": "isotope ratio mass spectrometers"
+ },
+ {
+ "const": "LBTG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LBTG/",
+ "title": "labelling tags"
+ },
+ {
+ "const": "LAB33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB33/",
+ "title": "laboratory autosamplers"
+ },
+ {
+ "const": "LAB42",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB42/",
+ "title": "laboratory optical rangefinders"
+ },
+ {
+ "const": "310",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/310/",
+ "title": "laser altimeters"
+ },
+ {
+ "const": "LAB21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB21/",
+ "title": "liquid scintillation counters"
+ },
+ {
+ "const": "116",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/116/",
+ "title": "lowered current profilers"
+ },
+ {
+ "const": "LAB31",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB31/",
+ "title": "magnetic susceptibility systems"
+ },
+ {
+ "const": "159",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/159/",
+ "title": "magnetometers"
+ },
+ {
+ "const": "90",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/90/",
+ "title": "manual biota samplers"
+ },
+ {
+ "const": "57",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/57/",
+ "title": "manual rock samplers"
+ },
+ {
+ "const": "MVBR",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MVBR/",
+ "title": "marine vibroseis"
+ },
+ {
+ "const": "LAB16",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB16/",
+ "title": "mass spectrometers"
+ },
+ {
+ "const": "383",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/383/",
+ "title": "metal analysers"
+ },
+ {
+ "const": "92",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/92/",
+ "title": "meteorological LiDARs"
+ },
+ {
+ "const": "102",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/102/",
+ "title": "meteorological packages"
+ },
+ {
+ "const": "184",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/184/",
+ "title": "microstructure sensors"
+ },
+ {
+ "const": "157",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/157/",
+ "title": "multi-beam echosounders"
+ },
+ {
+ "const": "154",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/154/",
+ "title": "multi-channel seismic reflection systems"
+ },
+ {
+ "const": "MMIS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/MMIS/",
+ "title": "multi-frame imaging sonar"
+ },
+ {
+ "const": "68",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/68/",
+ "title": "multinet"
+ },
+ {
+ "const": "NATS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/NATS/",
+ "title": "natural source"
+ },
+ {
+ "const": "69",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/69/",
+ "title": "neuston net"
+ },
+ {
+ "const": "LAB18",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB18/",
+ "title": "nuclear magnetic resonance spectrometers"
+ },
+ {
+ "const": "181",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/181/",
+ "title": "nutrient analysers"
+ },
+ {
+ "const": "91",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/91/",
+ "title": "observers"
+ },
+ {
+ "const": "304",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/304/",
+ "title": "ocean colour radiometers"
+ },
+ {
+ "const": "123",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/123/",
+ "title": "optical backscatter sensors"
+ },
+ {
+ "const": "LAB05",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB05/",
+ "title": "optical microscopes"
+ },
+ {
+ "const": "355",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/355/",
+ "title": "pH sensors"
+ },
+ {
+ "const": "32",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/32/",
+ "title": "particulates samplers"
+ },
+ {
+ "const": "63",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/63/",
+ "title": "passive fishing gear"
+ },
+ {
+ "const": "23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/23/",
+ "title": "pelagic trawl nets"
+ },
+ {
+ "const": "361",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/361/",
+ "title": "penetrometers"
+ },
+ {
+ "const": "360",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/360/",
+ "title": "piezometers"
+ },
+ {
+ "const": "352",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/352/",
+ "title": "plankton categorisers and counters"
+ },
+ {
+ "const": "22",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/22/",
+ "title": "plankton nets"
+ },
+ {
+ "const": "21",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/21/",
+ "title": "plankton recorders"
+ },
+ {
+ "const": "LAB43",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB43/",
+ "title": "plate readers"
+ },
+ {
+ "const": "385",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/385/",
+ "title": "platform attitude sensors"
+ },
+ {
+ "const": "359",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/359/",
+ "title": "plummets"
+ },
+ {
+ "const": "381",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/381/",
+ "title": "precipitation gauges"
+ },
+ {
+ "const": "14",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/14/",
+ "title": "precipitation samplers"
+ },
+ {
+ "const": "LAB46",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB46/",
+ "title": "proton transfer reaction mass spectrometers"
+ },
+ {
+ "const": "186",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/186/",
+ "title": "radar altimeters"
+ },
+ {
+ "const": "POS23",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/POS23/",
+ "title": "radio tracking systems"
+ },
+ {
+ "const": "122",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/122/",
+ "title": "radiometers"
+ },
+ {
+ "const": "103",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/103/",
+ "title": "radiosondes"
+ },
+ {
+ "const": "356",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/356/",
+ "title": "redox potential sensors"
+ },
+ {
+ "const": "LAB44",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB44/",
+ "title": "refractometers"
+ },
+ {
+ "const": "53",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/53/",
+ "title": "rock corers"
+ },
+ {
+ "const": "54",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/54/",
+ "title": "rock dredges"
+ },
+ {
+ "const": "LAB29",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB29/",
+ "title": "rulers"
+ },
+ {
+ "const": "350",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/350/",
+ "title": "salinity sensor"
+ },
+ {
+ "const": "LAB30",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB30/",
+ "title": "salinometers"
+ },
+ {
+ "const": "59",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/59/",
+ "title": "sampling extent markers"
+ },
+ {
+ "const": "301",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/301/",
+ "title": "satellite positioning systems"
+ },
+ {
+ "const": "121",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/121/",
+ "title": "satellite tracking system"
+ },
+ {
+ "const": "305",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/305/",
+ "title": "scatterometers"
+ },
+ {
+ "const": "111",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/111/",
+ "title": "sea level recorders"
+ },
+ {
+ "const": "60",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/60/",
+ "title": "sediment dredges"
+ },
+ {
+ "const": "50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/50/",
+ "title": "sediment grabs"
+ },
+ {
+ "const": "52",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/52/",
+ "title": "sediment porewater samplers"
+ },
+ {
+ "const": "378",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/378/",
+ "title": "sediment profile imagers"
+ },
+ {
+ "const": "55",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/55/",
+ "title": "sediment settling tubes"
+ },
+ {
+ "const": "391",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/391/",
+ "title": "sediment suction samplers"
+ },
+ {
+ "const": "376",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/376/",
+ "title": "sediment surface markers"
+ },
+ {
+ "const": "33",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/33/",
+ "title": "sediment traps"
+ },
+ {
+ "const": "155",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/155/",
+ "title": "seismic refraction systems"
+ },
+ {
+ "const": "368",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/368/",
+ "title": "seismometers"
+ },
+ {
+ "const": "81",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/81/",
+ "title": "shipboard incubators"
+ },
+ {
+ "const": "152",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/152/",
+ "title": "sidescan sonars"
+ },
+ {
+ "const": "84",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/84/",
+ "title": "sieves and filters"
+ },
+ {
+ "const": "156",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/156/",
+ "title": "single-beam echosounders"
+ },
+ {
+ "const": "SBAG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SBAG/",
+ "title": "single-bubble airgun"
+ },
+ {
+ "const": "153",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/153/",
+ "title": "single-channel seismic reflection systems"
+ },
+ {
+ "const": "56",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/56/",
+ "title": "snow and ice samplers"
+ },
+ {
+ "const": "185",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/185/",
+ "title": "sound velocity sensors"
+ },
+ {
+ "const": "LAB20",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB20/",
+ "title": "spectrophotometers"
+ },
+ {
+ "const": "380",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/380/",
+ "title": "submarine cables"
+ },
+ {
+ "const": "303",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/303/",
+ "title": "surface current radars"
+ },
+ {
+ "const": "306",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/306/",
+ "title": "synthetic aperture radars"
+ },
+ {
+ "const": "309",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/309/",
+ "title": "terrestrial radiometers"
+ },
+ {
+ "const": "LAB28",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB28/",
+ "title": "thermal conductivimeters"
+ },
+ {
+ "const": "LAB50",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB50/",
+ "title": "thermal cyclers"
+ },
+ {
+ "const": "LAB47",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB47/",
+ "title": "thermal ionisation mass spectrometers"
+ },
+ {
+ "const": "135",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/135/",
+ "title": "thermistor chains"
+ },
+ {
+ "const": "133",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/133/",
+ "title": "thermosalinographs"
+ },
+ {
+ "const": "TFSAMP",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/TFSAMP/",
+ "title": "thin film metal samplers"
+ },
+ {
+ "const": "LAB12",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB12/",
+ "title": "titrators"
+ },
+ {
+ "const": "TRTG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/TRTG/",
+ "title": "tracking tags"
+ },
+ {
+ "const": "124",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/124/",
+ "title": "transmissometers"
+ },
+ {
+ "const": "51",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/51/",
+ "title": "unconsolidated sediment corers"
+ },
+ {
+ "const": "999",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/999/",
+ "title": "unknown"
+ },
+ {
+ "const": "LAB35",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/LAB35/",
+ "title": "voltammetry analysers"
+ },
+ {
+ "const": "302",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/302/",
+ "title": "water body temperature sensor"
+ },
+ {
+ "const": "377",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/377/",
+ "title": "water level markers"
+ },
+ {
+ "const": "WPS",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/WPS/",
+ "title": "water pressure sensors"
+ },
+ {
+ "const": "134",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/134/",
+ "title": "water temperature sensor"
+ },
+ {
+ "const": "SNWG",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/SNWG/",
+ "title": "watergun"
+ },
+ {
+ "const": "110",
+ "uri": "http://vocab.nerc.ac.uk/collection/L05/current/110/",
+ "title": "wave recorders"
+ }
+ ]
+ },
+ "meta": {
+ "description": "Data provider specific metadata pertaining to the observation",
+ "type": [
+ "string",
+ "object"
+ ]
+ }
+ },
+ "required": [
+ "time",
+ "latitude",
+ "longitude",
+ "depth"
+ ],
+ "minProperties": 5
+ }
+ }
+}
\ No newline at end of file
diff --git a/parquet_flask/__init__.py b/parquet_flask/__init__.py
new file mode 100644
index 0000000..584e842
--- /dev/null
+++ b/parquet_flask/__init__.py
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from flask import Flask
+from .v1 import blueprint
+
+
+def get_app():
+ app = Flask(__name__)
+ app.register_blueprint(blueprint)
+ # api.init_app(app)
+ # CORS(app)
+ return app
diff --git a/parquet_flask/aws/__init__.py b/parquet_flask/aws/__init__.py
new file mode 100644
index 0000000..fa8ccd5
--- /dev/null
+++ b/parquet_flask/aws/__init__.py
@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
\ No newline at end of file
diff --git a/parquet_flask/aws/aws_cred.py b/parquet_flask/aws/aws_cred.py
new file mode 100644
index 0000000..e5ea0ec
--- /dev/null
+++ b/parquet_flask/aws/aws_cred.py
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from parquet_flask.utils.config import Config
+
+
+class AwsCred:
+ def __init__(self):
+ self.__boto3_session = {
+ 'aws_access_key_id': Config().get_value('aws_access_key_id'),
+ 'aws_secret_access_key': Config().get_value('aws_secret_access_key'),
+ 'region_name': 'us-west-2',
+ }
+ aws_session_token = Config().get_value('aws_session_token')
+ if aws_session_token is not None:
+ self.__boto3_session['aws_session_token'] = aws_session_token
+
+ @property
+ def boto3_session(self):
+ return self.__boto3_session
+
+ @boto3_session.setter
+ def boto3_session(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__boto3_session = val
+ return
diff --git a/parquet_flask/aws/aws_ddb.py b/parquet_flask/aws/aws_ddb.py
new file mode 100644
index 0000000..f2bf1f8
--- /dev/null
+++ b/parquet_flask/aws/aws_ddb.py
@@ -0,0 +1,335 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import decimal
+import logging
+
+import boto3
+from boto3.dynamodb.conditions import Attr
+
+from parquet_flask.aws.aws_cred import AwsCred
+
+LOGGER = logging.getLogger(__name__)
+
+VALID_KEY_TYPE = ['S', 'N', 'B']
+
+
+class AwsDdbProps:
+ def __init__(self):
+ self.__tbl_name = None
+ self.__hash_key = None
+ self.__range_key = None
+ self.__hash_key_type = 'S'
+ self.__range_key_type = 'S'
+
+ @property
+ def tbl_name(self):
+ return self.__tbl_name
+
+ @tbl_name.setter
+ def tbl_name(self, val):
+ """
+ :param val: str
+ :return: None
+ """
+ self.__tbl_name = val
+ return
+
+ @property
+ def hash_key(self):
+ return self.__hash_key
+
+ @hash_key.setter
+ def hash_key(self, val):
+ """
+ :param val: str
+ :return: None
+ """
+ self.__hash_key = val
+ return
+
+ @property
+ def range_key(self):
+ return self.__range_key
+
+ @range_key.setter
+ def range_key(self, val):
+ """
+ :param val: str
+ :return: None
+ """
+ self.__range_key = val
+ return
+
+ @property
+ def hash_key_type(self):
+ return self.__hash_key_type
+
+ @hash_key_type.setter
+ def hash_key_type(self, val):
+ """
+ :param val: str - 'S', 'N', or 'B'
+ :return: None
+ """
+ if val not in VALID_KEY_TYPE:
+ raise ValueError('input is not valid type. {} vs. {}'.format(val, VALID_KEY_TYPE))
+ self.__hash_key_type = val
+ return
+
+ @property
+ def range_key_type(self):
+ return self.__range_key_type
+
+ @range_key_type.setter
+ def range_key_type(self, val):
+ """
+ :param val: str - 'S', 'N', or 'B'
+ :return: None
+ """
+ if val not in VALID_KEY_TYPE:
+ raise ValueError('input is not valid type. {} vs. {}'.format(val, VALID_KEY_TYPE))
+ self.__range_key_type = val
+ return
+
+
+class AwsDdb(AwsCred):
+ def __init__(self, props=AwsDdbProps()):
+ super().__init__()
+ self.__props = props
+ self._ddb_client = boto3.Session(**self.boto3_session).client('dynamodb')
+ self._ddb_resource = boto3.Session(**self.boto3_session).resource('dynamodb')
+
+ def has_table(self):
+ if self.__props.tbl_name is None:
+ raise ValueError('missing tbl_name')
+ try:
+ tbl_details = self._ddb_client.describe_table(TableName=self.__props.tbl_name)
+ return tbl_details
+ except Exception as e:
+ # TODO should check if exception is this one "ResourceNotFoundException". if not, throw the error
+ return None
+
+ def create_table(self, gsi_list=[]):
+ """
+ ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.create_table
+ :param gsi_list: list - [{'IndexName': 'string','KeySchema': [{'AttributeName': 'string','KeyType': 'HASH'|'RANGE'},]}]
+ :param primary_key: str - Hash Key
+ :param secondary_key: str - Range Key (optional)
+ :param primary_key_type: str - 'S', 'N', or 'B'
+ :param secondary_key_type: str - 'S', 'N', or 'B'
+ :return: dict - create table result
+ """
+ if self.__props.tbl_name is None:
+ raise ValueError('missing tbl_name')
+ if self.__props.hash_key is None:
+ raise ValueError('missing hash_key')
+ LOGGER.info('creating a table: {}'.format(self.__props.tbl_name))
+ attribute_definitions = [
+ {
+ 'AttributeName': self.__props.hash_key,
+ 'AttributeType': self.__props.hash_key_type,
+ }
+ ]
+ key_schema = [
+ {
+ 'AttributeName': self.__props.hash_key,
+ 'KeyType': 'HASH', # 'RANGE' if there is secondary key
+ }
+ ]
+ for each in gsi_list:
+ each['Projection'] = {'ProjectionType': 'ALL'}
+ if self.__props.range_key is not None:
+ attribute_definitions.append({
+ 'AttributeName': self.__props.range_key,
+ 'AttributeType': self.__props.range_key_type,
+ })
+ key_schema.append({
+ 'AttributeName': self.__props.range_key,
+ 'KeyType': 'RANGE',
+ })
+ create_tbl_params = {
+ 'TableName': self.__props.tbl_name,
+ 'AttributeDefinitions': attribute_definitions,
+ 'KeySchema': key_schema,
+ 'BillingMode': 'PAY_PER_REQUEST', # TODO setting it to on-demand. might need to re-visit later
+ 'SSESpecification': {'Enabled': False} # TODO had to disable it since it does not support 'AES256' yet.
+ }
+ if len(gsi_list) > 0:
+ create_tbl_params['GlobalSecondaryIndexes'] = gsi_list
+ create_result = self._ddb_client.create_table(**create_tbl_params)
+ return create_result
+
+ def _replace_decimals(self, obj):
+ """
+ Ref:
+ https://stackoverflow.com/a/46738251 in the comments
+ https://github.com/boto/boto3/issues/369#issuecomment-157205696
+
+ :param obj:
+ :return:
+ """
+ if isinstance(obj, list):
+ for i in range(len(obj)):
+ obj[i] = self._replace_decimals(obj[i])
+ return obj
+ elif isinstance(obj, dict):
+ for k in obj.keys():
+ obj[k] = self._replace_decimals(obj[k])
+ return obj
+ elif isinstance(obj, decimal.Decimal):
+ if obj % 1 == 0:
+ return int(obj)
+ else:
+ return float(obj)
+ else:
+ return obj
+
+ def get_one_item(self, hash_val, range_val=None):
+ """
+ retrieving a single item based on hash key
+ :param hash_val:
+ :param range_val:
+ :return:
+ """
+ LOGGER.info('retrieving one item from DDB using they key')
+ query_key = {self.__props.hash_key: hash_val}
+ if range_val is not None and self.__props.range_key is not None:
+ query_key[self.__props.range_key] = range_val
+ item_result = self._ddb_resource.Table(self.__props.tbl_name).get_item(
+ Key=query_key
+ )
+ if 'Item' not in item_result:
+ return None
+ return self._replace_decimals(item_result['Item'])
+
+ def delete_one_item(self, hash_val, range_val=None):
+ """
+
+ Sample Response:
+ {'Attributes': {...}}
+ {'RequestId': '70PUK7HSNQI6VLHRM1Q7VPESJ3VV4KQNSO5AEMVJF66Q9ASUAAJG', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Mon, 08 Mar 2021 18:04:35 GMT', 'content-type': 'application/x-amz-json-1.0', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '70PUK7HSNQI6VLHRM1Q7VPESJ3VV4KQNSO5AEMVJF66Q9ASUAAJG', 'x-amz-crc32': '2745614147'}, 'RetryAttempts': 0}
+
+ :param hash_val:
+ :param range_val:
+ :return:
+ """
+ LOGGER.info('deleting one item from DDB using they key')
+ query_key = {self.__props.hash_key: hash_val}
+ if range_val is not None and self.__props.range_key is not None:
+ query_key[self.__props.range_key] = range_val
+ item_result = self._ddb_resource.Table(self.__props.tbl_name).delete_item(Key=query_key, ReturnValues='ALL_OLD')
+ if 'Attributes' not in item_result:
+ LOGGER.warning('cannot retrieved deleted attributes.')
+ return None
+ return item_result['Attributes']
+
+ def add_one_item(self, item_dict, hash_val, range_val=None, replace=False):
+ LOGGER.info('adding one item from DDB using they key')
+ item_dict[self.__props.hash_key] = hash_val
+ if range_val is not None and self.__props.range_key is not None:
+ item_dict[self.__props.range_key] = range_val
+
+ addition_arguments = {
+ 'Item': item_dict,
+ 'ReturnValues': 'ALL_OLD',
+ }
+ if replace is True:
+ if range_val is not None and self.__props.range_key is not None:
+ condition = Attr(self.__props.hash_key).eq(hash_val) and Attr(self.__props.range_key).ne(range_val)
+ else:
+ condition = Attr(self.__props.hash_key).eq(hash_val)
+ else:
+ if range_val is not None and self.__props.range_key is not None:
+ condition = Attr(self.__props.hash_key).ne(hash_val) and Attr(self.__props.range_key).ne(range_val)
+ else:
+ condition = Attr(self.__props.hash_key).ne(hash_val)
+ addition_arguments['ConditionExpression'] = condition
+ item_result = self._ddb_resource.Table(self.__props.tbl_name).put_item(**addition_arguments)
+ """
+ {'ResponseMetadata': {'RequestId': '49876A3IFHPMRFIEUMANGFAO8VVV4KQNSO5AEMVJF66Q9ASUAAJG', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Mon, 08 Mar 2021 17:58:08 GMT', 'content-type': 'application/x-amz-json-1.0', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '49876A3IFHPMRFIEUMANGFAO8VVV4KQNSO5AEMVJF66Q9ASUAAJG', 'x-amz-crc32': '2745614147'}, 'RetryAttempts': 0}}
+ """
+ # TODO check result
+ return
+
+ def scan_tbl(self, conditions_dict):
+ LOGGER.info('scanning items from DDB using they key')
+ current_tbl = self._ddb_resource.Table(self.__props.tbl_name)
+ item_result = current_tbl.scan(
+ Limit=1,
+ ScanFilter=conditions_dict,
+ Select='ALL_ATTRIBUTES')
+ all_results = item_result['Items']
+ while 'LastEvaluatedKey' in item_result and item_result['LastEvaluatedKey'] is not None: # pagination
+ item_result = current_tbl.scan(
+ Limit=100,
+ ScanFilter=conditions_dict,
+ ExclusiveStartKey=item_result['LastEvaluatedKey'],
+ Select='ALL_ATTRIBUTES')
+ all_results.extend(item_result['Items'])
+ return self._replace_decimals(all_results)
+
+ def update_one_item(self, update_expression, expression_names, expression_vals, hash_val, range_val=None, retrieve_new_val=True):
+ """
+ Usage : increment or decrement
+
+ddb.update_one_item('SET #created_at_key = #created_at_key + :created_at_val', {'#created_at_key': 'created_at'}, {':created_at_val': -50}, '61725b56-3016-42c6-9006-c0b5d9017fee')
+
+ :param update_expression: str - example: add #created_key = :created_val
+ :param expression_names: dict - {'#created_key': 'created_at'}
+ :param expression_vals: dict - {':created_val': 123}
+ :param hash_val: str
+ :param range_val: str
+ :return:
+ """
+ LOGGER.info('updating one item from DDB using they key')
+ query_key = {self.__props.hash_key: hash_val}
+ if range_val is not None and self.__props.range_key is not None:
+ query_key[self.__props.range_key] = range_val
+ item_result = self._ddb_resource.Table(self.__props.tbl_name).update_item(
+ Key=query_key,
+ UpdateExpression=update_expression,
+ ExpressionAttributeNames=expression_names,
+ ExpressionAttributeValues=expression_vals,
+ ReturnValues='ALL_NEW' if retrieve_new_val is True else 'ALL_OLD'
+ )
+ if 'Attributes' not in item_result:
+ return None
+ return self._replace_decimals(item_result['Attributes'])
+
+ def __get_ddb_type(self, input_val):
+ if isinstance(input_val, str):
+ return 'S'
+ if isinstance(input_val, bool):
+ return 'B'
+ return 'N'
+
+ def get_from_index(self, index_name: str, hash_dict: dict):
+ """
+ https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.query
+ :param index_name: str - name of a secondary index
+ :param hash_dict: dictionary of {'name': 'value'} of a hash key. Only 1 is allowed
+ :return:
+ """
+ hash_val = [v for v in hash_dict.values()][0]
+ query_dict = {
+ 'IndexName': index_name,
+ 'Select': 'ALL_ATTRIBUTES', # 'ALL_ATTRIBUTES'|'ALL_PROJECTED_ATTRIBUTES'|'SPECIFIC_ATTRIBUTES'|'COUNT'
+ 'Limit': 1,
+ 'ConsistentRead': False,
+ 'KeyConditionExpression': boto3.dynamodb.conditions.Key([k for k in hash_dict.keys()][0]).between(hash_val),
+ }
+ item_result = self._ddb_resource.Table(self.__props.tbl_name).query(**query_dict)
+ updated_result = [self._replace_decimals(k) for k in item_result['Items']]
+ return updated_result
diff --git a/parquet_flask/aws/aws_s3.py b/parquet_flask/aws/aws_s3.py
new file mode 100644
index 0000000..08b5f99
--- /dev/null
+++ b/parquet_flask/aws/aws_s3.py
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import os
+
+import boto3
+
+from parquet_flask.aws.aws_cred import AwsCred
+from parquet_flask.utils.file_utils import FileUtils
+
+LOGGER = logging.getLogger(__name__)
+
+
+class AwsS3(AwsCred):
+ def __init__(self):
+ super().__init__()
+ self.__valid_s3_schemas = ['s3://', 's3a://', 's3s://']
+ self.__s3_client = boto3.Session(**self.boto3_session).client('s3')
+ self.__target_bucket = None
+ self.__target_key = None
+
+ def __get_all_s3_files_under(self, bucket, prefix, with_versions=False):
+ list_method_name = 'list_object_versions' if with_versions is True else 'list_objects_v2'
+ page_key = 'Versions' if with_versions is True else 'Contents'
+ paginator = self.__s3_client.get_paginator(list_method_name)
+ operation_parameters = {
+ 'Bucket': bucket,
+ 'Prefix': prefix
+ }
+ page_iterator = paginator.paginate(**operation_parameters)
+ for eachPage in page_iterator:
+ if page_key not in eachPage:
+ continue
+ for fileObj in eachPage[page_key]:
+ yield fileObj
+
+ def get_child_s3_files(self, bucket, prefix, additional_checks=lambda x: True, with_versions=False):
+ for fileObj in self.__get_all_s3_files_under(bucket, prefix, with_versions=with_versions):
+ if additional_checks(fileObj):
+ yield fileObj['Key'], fileObj['Size']
+
+ def set_s3_url(self, s3_url):
+ LOGGER.debug(f'setting s3_url: {s3_url}')
+ self.__target_bucket, self.__target_key = self.split_s3_url(s3_url)
+ LOGGER.debug(f'props: {self.__target_bucket}, {self.__target_key}')
+ return self
+
+ def split_s3_url(self, s3_url):
+ s3_schema = [k for k in self.__valid_s3_schemas if s3_url.startswith(k)]
+ if len(s3_schema) != 1:
+ raise ValueError('invalid s3 url: {}'.format(s3_url))
+
+ s3_schema_length = len(s3_schema[0])
+ split_index = s3_url[s3_schema_length:].find('/')
+ bucket = s3_url[s3_schema_length: split_index+s3_schema_length]
+ key = s3_url[(split_index + s3_schema_length + 1):]
+ return bucket, key
+
+ def __tag_existing_obj(self, other_tags={}):
+ if len(other_tags) == 0:
+ return
+ tags = {
+ 'TagSet': []
+ }
+ for key, val in other_tags.items():
+ tags['TagSet'].append({
+ 'Key': key,
+ 'Value': str(val)
+ })
+ self.__s3_client.put_object_tagging(Bucket=self.__target_bucket, Key=self.__target_key, Tagging=tags)
+ return
+
+ def add_tags_to_obj(self, other_tags={}):
+ """
+ retrieve existing tags first and append new tags to them
+
+ :param bucket: string
+ :param s3_key: string
+ :param other_tags: dict
+ :return: bool
+ """
+ if len(other_tags) == 0:
+ return False
+ response = self.__s3_client.get_object_tagging(Bucket=self.__target_bucket, Key=self.__target_key)
+ if 'TagSet' not in response:
+ return False
+ all_tags = {k['Key']: k['Value'] for k in response['TagSet']}
+ for k, v in other_tags.items():
+ all_tags[k] = v
+ pass
+ self.__tag_existing_obj(all_tags)
+ return True
+
+ def download(self, local_dir, file_name=None):
+ if not FileUtils.dir_exist(local_dir):
+ raise ValueError('missing directory')
+ if file_name is None:
+ LOGGER.debug(f'setting the downloading filename from target_key: {self.__target_key}')
+ file_name = os.path.basename(self.__target_key)
+ local_file_path = os.path.join(local_dir, file_name)
+ LOGGER.debug(f'downloading to local_file_path: {local_file_path}')
+ self.__s3_client.download_file(self.__target_bucket, self.__target_key, local_file_path)
+ LOGGER.debug(f'file downloaded')
+ return local_file_path
diff --git a/parquet_flask/io_logic/__init__.py b/parquet_flask/io_logic/__init__.py
new file mode 100644
index 0000000..fa8ccd5
--- /dev/null
+++ b/parquet_flask/io_logic/__init__.py
@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
\ No newline at end of file
diff --git a/parquet_flask/io_logic/cdms_constants.py b/parquet_flask/io_logic/cdms_constants.py
new file mode 100644
index 0000000..c39b047
--- /dev/null
+++ b/parquet_flask/io_logic/cdms_constants.py
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class CDMSConstants:
+ time_col = 'time'
+ provider_col = 'provider'
+ project_col = 'project'
+ platform_code_col = 'platform_code'
+ platform_col = 'platform'
+ code_col = 'code'
+ job_id_col = 'job_id'
+ time_obj_col = 'time_obj'
+ year_col = 'year'
+ month_col = 'month'
+ observations_key = 'observations'
+ lat_col = 'latitude'
+ lon_col = 'longitude'
+ depth_col = 'depth'
+
+ s3_url_key = 's3_url'
+ uuid_key = 'uuid'
+ ingested_date_key = 'ingested_date'
+ checksum_key = 'checksum'
+ file_size_key = 'file_size'
+ records_count_key = 'records_count'
+ job_start_key = 'job_start_time'
+ job_end_key = 'job_end_time'
diff --git a/parquet_flask/io_logic/ingest_new_file.py b/parquet_flask/io_logic/ingest_new_file.py
new file mode 100644
index 0000000..7724d11
--- /dev/null
+++ b/parquet_flask/io_logic/ingest_new_file.py
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.io_logic.retrieve_spark_session import RetrieveSparkSession
+from parquet_flask.io_logic.sanitize_record import SanitizeRecord
+from parquet_flask.utils.config import Config
+from parquet_flask.utils.file_utils import FileUtils
+
+from pyspark.sql.functions import to_timestamp, year, month, lit
+
+LOGGER = logging.getLogger(__name__)
+
+
+class IngestNewJsonFile:
+ def __init__(self):
+ self.__sss = RetrieveSparkSession()
+ config = Config()
+ self.__app_name = config.get_value('spark_app_name')
+ self.__master_spark = config.get_value('master_spark_url')
+ self.__mode = 'append'
+ self.__parquet_name = config.get_value('parquet_file_name')
+
+ @staticmethod
+ def create_df(spark_session, data_list, job_id, provider, project):
+ LOGGER.debug(f'creating data frame with length {len(data_list)}')
+ df = spark_session.createDataFrame(data_list)
+ LOGGER.debug(f'adding columns')
+ df = df.withColumn(CDMSConstants.time_obj_col, to_timestamp(CDMSConstants.time_col))\
+ .withColumn(CDMSConstants.year_col, year(CDMSConstants.time_col))\
+ .withColumn(CDMSConstants.month_col, month(CDMSConstants.time_col))\
+ .withColumn(CDMSConstants.platform_code_col, df[CDMSConstants.platform_col][CDMSConstants.code_col])\
+ .withColumn(CDMSConstants.job_id_col, lit(job_id))\
+ .withColumn(CDMSConstants.provider_col, lit(provider))\
+ .withColumn(CDMSConstants.project_col, lit(project))
+ # .withColumn('ingested_date', lit(TimeUtils.get_current_time_str()))
+ LOGGER.debug(f'create writer')
+ df_writer = df.write
+ all_partitions = [CDMSConstants.provider_col, CDMSConstants.project_col, CDMSConstants.platform_code_col,
+ CDMSConstants.year_col, CDMSConstants.month_col, CDMSConstants.job_id_col]
+ LOGGER.debug(f'create partitions')
+ df_writer = df_writer.partitionBy(all_partitions)
+ LOGGER.debug(f'created partitions')
+ return df_writer
+
+ def ingest(self, abs_file_path, job_id):
+ """
+ This method will assume that incoming file has data with in_situ_schema file.
+
+ So, it will definitely has `time`, `project`, and `provider`.
+
+ :param abs_file_path:
+ :param job_id:
+ :return: int - number of records
+ """
+ if not FileUtils.file_exist(abs_file_path):
+ raise ValueError('missing file to ingest it. path: {}'.format(abs_file_path))
+ LOGGER.debug(f'sanitizing the files')
+ input_json = SanitizeRecord(Config().get_value('in_situ_schema')).start(abs_file_path)
+ df_writer = self.create_df(self.__sss.retrieve_spark_session(self.__app_name, self.__master_spark),
+ input_json[CDMSConstants.observations_key],
+ job_id,
+ input_json[CDMSConstants.provider_col],
+ input_json[CDMSConstants.project_col])
+ df_writer.mode(self.__mode).parquet(self.__parquet_name, compression='GZIP')
+ LOGGER.debug(f'finished writing parquet')
+ return len(input_json[CDMSConstants.observations_key])
diff --git a/parquet_flask/io_logic/metadata_tbl_interface.py b/parquet_flask/io_logic/metadata_tbl_interface.py
new file mode 100644
index 0000000..e481305
--- /dev/null
+++ b/parquet_flask/io_logic/metadata_tbl_interface.py
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import abc
+
+
+class MetadataTblInterface(metaclass=abc.ABCMeta):
+ @abc.abstractmethod
+ def insert_record(self, new_record):
+ return
+
+ @abc.abstractmethod
+ def replace_record(self, new_record):
+ return
+
+ @abc.abstractmethod
+ def get_by_s3_url(self, s3_url):
+ return
+
+ @abc.abstractmethod
+ def get_by_uuid(self, uuid):
+ return
+
+ @abc.abstractmethod
+ def query_by_date_range(self, start_time, end_time):
+ return
diff --git a/parquet_flask/io_logic/metadata_tbl_io.py b/parquet_flask/io_logic/metadata_tbl_io.py
new file mode 100644
index 0000000..61ae049
--- /dev/null
+++ b/parquet_flask/io_logic/metadata_tbl_io.py
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from parquet_flask.aws.aws_ddb import AwsDdb, AwsDdbProps
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.io_logic.metadata_tbl_interface import MetadataTblInterface
+
+
+class MetadataTblIO(MetadataTblInterface):
+ """
+ Table columns
+ - s3_url
+ - uuid
+ - size
+ - ingested_date
+ - md5
+ - num-of-records
+
+ Table settings
+ s3_url as primary key
+ secondary index: uuid
+ """
+ def __init__(self):
+ ddb_props = AwsDdbProps()
+ ddb_props.hash_key = CDMSConstants.s3_url_key
+ ddb_props.tbl_name = 'cdms_parquet_meta_dev_v1' # TODO come from config
+ self.__uuid_index = 'uuid-index'
+ self.__ddb = AwsDdb(ddb_props)
+
+ def insert_record(self, new_record):
+ self.__ddb.add_one_item(new_record, new_record[CDMSConstants.s3_url_key])
+ return
+
+ def replace_record(self, new_record):
+ self.__ddb.add_one_item(new_record, new_record[CDMSConstants.s3_url_key], replace=True)
+ return
+
+ def get_by_s3_url(self, s3_url):
+ return self.__ddb.get_one_item(s3_url)
+
+ def get_by_uuid(self, uuid):
+ return self.__ddb.get_from_index(self.__uuid_index, {CDMSConstants.uuid_key: uuid})
+
+ def query_by_date_range(self, start_time, end_time):
+ raise NotImplementedError('cannot implement range query to the primary key in DDB')
diff --git a/parquet_flask/io_logic/query.py b/parquet_flask/io_logic/query.py
new file mode 100644
index 0000000..2e65921
--- /dev/null
+++ b/parquet_flask/io_logic/query.py
@@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from parquet_flask.io_logic.retrieve_spark_session import RetrieveSparkSession
+from parquet_flask.utils.config import Config
+
+LOGGER = logging.getLogger(__name__)
+
+
+class QueryParquet:
+ def __init__(self):
+ self.__sss = RetrieveSparkSession()
+ config = Config()
+ self.__app_name = config.get_value('spark_app_name')
+ self.__master_spark = config.get_value('master_spark_url')
+ self.__parquet_name = config.get_value('parquet_file_name')
+ self.__depth = None
+ self.__min_time = None
+ self.__max_time = None
+ self.__min_lat_lon = None
+ self.__max_lat_lon = None
+ self.__limit = 0
+ self.__size = 1000
+
+ @property
+ def limit(self):
+ return self.__limit
+
+ @limit.setter
+ def limit(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__limit = val
+ return
+
+ @property
+ def size(self):
+ return self.__size
+
+ @size.setter
+ def size(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__size = val
+ return
+
+ @property
+ def depth(self):
+ return self.__depth
+
+ @depth.setter
+ def depth(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__depth = val
+ return
+
+ @property
+ def min_time(self):
+ return self.__min_time
+
+ @min_time.setter
+ def min_time(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__min_time = val
+ return
+
+ @property
+ def max_time(self):
+ return self.__max_time
+
+ @max_time.setter
+ def max_time(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__max_time = val
+ return
+
+ @property
+ def min_lat_lon(self):
+ return self.__min_lat_lon
+
+ @min_lat_lon.setter
+ def min_lat_lon(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__min_lat_lon = val
+ return
+
+ @property
+ def max_lat_lon(self):
+ return self.__max_lat_lon
+
+ @max_lat_lon.setter
+ def max_lat_lon(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__max_lat_lon = val
+ return
+
+ def __construct_query(self):
+ conditions = []
+ if self.depth is not None:
+ conditions.append(f'depth >= {self.depth}')
+ if self.min_time is not None:
+ conditions.append(f'time_obj >= {self.min_time}')
+ if self.max_time is not None:
+ conditions.append(f'time_obj <= {self.max_time}')
+ if self.min_lat_lon is not None:
+ conditions.append(f'latitude >= {self.min_lat_lon[0]}')
+ conditions.append(f'longitude >= {self.min_lat_lon[1]}')
+ if self.max_lat_lon is not None:
+ conditions.append(f'latitude <= {self.max_lat_lon[0]}')
+ conditions.append(f'longitude <= {self.max_lat_lon[1]}')
+ return conditions
+
+ def search(self):
+ conditions = self.__construct_query()
+ conditions = ' AND '.join(conditions)
+ sql_stmt = 'select * from ParquetTable '
+ if len(conditions) > 0:
+ sql_stmt = '{} where {} ; '.format(sql_stmt, conditions)
+ LOGGER.debug(f'query statement: {sql_stmt}')
+ spark = self.__sss.retrieve_spark_session(self.__app_name, self.__master_spark)
+ spark.read.parquet(self.__parquet_name).createOrReplaceTempView("parquetTable")
+ result = spark.sql(sql_stmt).limit(self.limit).collect()
+ LOGGER.debug(f'query result: {result}')
+ return [k.asDict() for k in result]
diff --git a/parquet_flask/io_logic/query_v2.py b/parquet_flask/io_logic/query_v2.py
new file mode 100644
index 0000000..b16cdda
--- /dev/null
+++ b/parquet_flask/io_logic/query_v2.py
@@ -0,0 +1,414 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from datetime import datetime
+
+# from pyspark import F
+from pyspark.sql.dataframe import DataFrame
+
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.utils.config import Config
+from parquet_flask.utils.time_utils import TimeUtils
+
+LOGGER = logging.getLogger(__name__)
+
+QUERY_PROPS_SCHEMA = {
+ 'type': 'object',
+ 'properties': {
+ 'start_from': {'type': 'integer'},
+ 'size': {'type': 'integer'},
+ 'columns': {
+ 'type': 'array',
+ 'items': {'type': 'string'},
+ 'minItems': 0,
+ },
+ 'platform_code': {'type': 'string'},
+ 'provider': {'type': 'string'},
+ 'project': {'type': 'string'},
+ 'min_depth': {'type': 'number'},
+ 'max_depth': {'type': 'number'},
+ 'min_time': {'type': 'string'},
+ 'max_time': {'type': 'string'},
+ 'min_lat_lon': {'type': 'array', 'items': {'type': 'number'}, 'minItems': 2, 'maxItems': 2},
+ 'max_lat_lon': {'type': 'array', 'items': {'type': 'number'}, 'minItems': 2, 'maxItems': 2},
+ },
+ 'required': ['start_from', 'size', 'min_depth', 'max_depth', 'min_time', 'max_time', 'min_lat_lon', 'max_lat_lon'],
+}
+
+
+class QueryProps:
+ def __init__(self):
+ self.__variable = None
+ self.__quality_flag = False
+ self.__platform_code = None
+ self.__project = None
+ self.__provider = None
+ self.__device = None
+ self.__min_depth = None
+ self.__max_depth = None
+ self.__min_datetime = None
+ self.__max_datetime = None
+ self.__min_lat_lon = None
+ self.__max_lat_lon = None
+ self.__start_at = 0
+ self.__size = 0
+ self.__columns = []
+
+ @property
+ def variable(self):
+ return self.__variable
+
+ @variable.setter
+ def variable(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__variable = val
+ return
+
+ @property
+ def quality_flag(self):
+ return self.__quality_flag
+
+ @quality_flag.setter
+ def quality_flag(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__quality_flag = val
+ return
+
+ @property
+ def platform_code(self):
+ return self.__platform_code
+
+ @platform_code.setter
+ def platform_code(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__platform_code = val
+ return
+
+ def from_json(self, input_json):
+ self.start_at = input_json['start_from']
+ self.size = input_json['size']
+ self.min_depth = input_json['min_depth']
+ self.max_depth = input_json['max_depth']
+ self.min_datetime = input_json['min_time']
+ self.max_datetime = input_json['max_time']
+ self.min_lat_lon = input_json['min_lat_lon']
+ self.max_lat_lon = input_json['max_lat_lon']
+ if 'project' in input_json:
+ self.project = input_json['project']
+ if 'provider' in input_json:
+ self.provider = input_json['provider']
+ if 'device' in input_json:
+ self.provider = input_json['device']
+ if 'platform_code' in input_json:
+ self.platform_code = input_json['platform_code']
+ if 'columns' in input_json:
+ self.columns = input_json['columns']
+ return self
+
+ @property
+ def project(self):
+ return self.__project
+
+ @project.setter
+ def project(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__project = val
+ return
+
+ @property
+ def provider(self):
+ return self.__provider
+
+ @provider.setter
+ def provider(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__provider = val
+ return
+
+ @property
+ def device(self):
+ return self.__device
+
+ @device.setter
+ def device(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__device = val
+ return
+
+ @property
+ def min_depth(self):
+ return self.__min_depth
+
+ @min_depth.setter
+ def min_depth(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__min_depth = val
+ return
+
+ @property
+ def max_depth(self):
+ return self.__max_depth
+
+ @max_depth.setter
+ def max_depth(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__max_depth = val
+ return
+
+ @property
+ def min_datetime(self):
+ return self.__min_datetime
+
+ @min_datetime.setter
+ def min_datetime(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__min_datetime = val
+ return
+
+ @property
+ def max_datetime(self):
+ return self.__max_datetime
+
+ @max_datetime.setter
+ def max_datetime(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__max_datetime = val
+ return
+
+ @property
+ def min_lat_lon(self):
+ return self.__min_lat_lon
+
+ @min_lat_lon.setter
+ def min_lat_lon(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__min_lat_lon = val
+ return
+
+ @property
+ def max_lat_lon(self):
+ return self.__max_lat_lon
+
+ @max_lat_lon.setter
+ def max_lat_lon(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__max_lat_lon = val
+ return
+
+ @property
+ def start_at(self):
+ return self.__start_at
+
+ @start_at.setter
+ def start_at(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__start_at = val
+ return
+
+ @property
+ def size(self):
+ return self.__size
+
+ @size.setter
+ def size(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__size = val
+ return
+
+ @property
+ def columns(self):
+ return self.__columns
+
+ @columns.setter
+ def columns(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__columns = val
+ return
+
+
+class Query:
+ def __init__(self, props=QueryProps()):
+ self.__props = props
+ config = Config()
+ self.__app_name = config.get_value('spark_app_name')
+ self.__master_spark = config.get_value('master_spark_url')
+ self.__parquet_name = config.get_value('parquet_file_name')
+
+ def __add_conditions(self):
+ conditions = []
+ min_year = None
+ max_year = None
+ if self.__props.platform_code is not None:
+ LOGGER.debug(f'setting platform_code condition: {self.__props.platform_code}')
+ conditions.append(f"{CDMSConstants.platform_code_col} == '{self.__props.platform_code}'")
+ if self.__props.provider is not None:
+ LOGGER.debug(f'setting provider condition: {self.__props.provider}')
+ conditions.append(f"{CDMSConstants.provider_col} == '{self.__props.provider}'")
+ if self.__props.project is not None:
+ LOGGER.debug(f'setting project condition: {self.__props.project}')
+ conditions.append(f"{CDMSConstants.project_col} == '{self.__props.project}'")
+ if self.__props.min_datetime is not None:
+ LOGGER.debug(f'setting datetime min condition: {self.__props.min_datetime}')
+ min_year = TimeUtils.get_datetime_obj(self.__props.min_datetime).year
+ conditions.append(f"{CDMSConstants.year_col} >= {min_year}")
+ conditions.append(f"{CDMSConstants.time_obj_col} >= '{self.__props.min_datetime}'")
+ if self.__props.max_datetime is not None:
+ LOGGER.debug(f'setting datetime max condition: {self.__props.max_datetime}')
+ max_year = TimeUtils.get_datetime_obj(self.__props.max_datetime).year
+ conditions.append(f"{CDMSConstants.year_col} <= {max_year}")
+ conditions.append(f"{CDMSConstants.time_obj_col} <= '{self.__props.max_datetime}'")
+ if min_year is not None and max_year is not None and min_year == max_year:
+ LOGGER.debug(f'setting month duration condition: {self.__props.max_datetime}')
+ conditions.append(f"{CDMSConstants.month_col} >= {TimeUtils.get_datetime_obj(self.__props.min_datetime).month}")
+ conditions.append(f"{CDMSConstants.month_col} <= {TimeUtils.get_datetime_obj(self.__props.max_datetime).month}")
+ if self.__props.min_lat_lon is not None:
+ LOGGER.debug(f'setting Lat-Lon min condition: {self.__props.min_lat_lon}')
+ conditions.append(f"{CDMSConstants.lat_col} >= {self.__props.min_lat_lon[0]}")
+ conditions.append(f"{CDMSConstants.lon_col} >= {self.__props.min_lat_lon[1]}")
+ if self.__props.max_lat_lon is not None:
+ LOGGER.debug(f'setting Lat-Lon max condition: {self.__props.max_lat_lon}')
+ conditions.append(f"{CDMSConstants.lat_col} <= {self.__props.max_lat_lon[0]}")
+ conditions.append(f"{CDMSConstants.lon_col} <= {self.__props.max_lat_lon[1]}")
+ if self.__props.min_depth is not None:
+ LOGGER.debug(f'setting depth min condition: {self.__props.min_depth}')
+ conditions.append(f"{CDMSConstants.depth_col} >= {self.__props.min_depth}")
+ if self.__props.max_depth is not None:
+ LOGGER.debug(f'setting depth max condition: {self.__props.max_depth}')
+ conditions.append(f"{CDMSConstants.depth_col} <= {self.__props.max_depth}")
+ if self.__props.variable is not None:
+ LOGGER.debug(f'setting not null variable: {self.__props.variable}')
+ conditions.append(f"{self.__props.variable} <= NULL")
+ self.__props.columns.append(self.__props.variable)
+ if self.__props.quality_flag is True:
+ LOGGER.debug(f'adding quality flag for : {self.__props.variable}')
+ self.__props.columns.append(f'{self.__props.variable}_quality')
+ LOGGER.debug(f'conditions list: {conditions}')
+ return ' AND '.join(conditions)
+
+ def __retrieve_spark(self):
+ from parquet_flask.io_logic.retrieve_spark_session import RetrieveSparkSession
+ spark = RetrieveSparkSession().retrieve_spark_session(self.__app_name, self.__master_spark)
+ return spark
+
+ def __sql_query(self, spark_session=None):
+ conditions = self.__add_conditions()
+ sql_stmt = 'select count(*) from ParquetTable '
+ if len(conditions) > 0:
+ sql_stmt = f'{sql_stmt} where {conditions} ; '
+ LOGGER.debug(f'query statement: {sql_stmt}')
+ time_start = datetime.now()
+ spark = self.__retrieve_spark() if spark_session is None else spark_session
+ spark.read.parquet(self.__parquet_name).createOrReplaceTempView("parquetTable")
+ read_df_time = datetime.now()
+ LOGGER.debug(f'query_2 parquet read created at {read_df_time}. took: {read_df_time - time_start}')
+ result_count = spark.sql(sql_stmt).collect()
+ time_end = datetime.now()
+ LOGGER.debug(f'query_2 count duration: {time_end - time_start}')
+
+ sql_stmt = 'select * from ParquetTable '
+ if len(conditions) > 0:
+ sql_stmt = f'{sql_stmt} where {conditions} ; ' # limit {self.__props.start_at + self.__props.size},{self.__props.size}
+ LOGGER.debug(f'query statement: {sql_stmt}')
+ removing_cols = [CDMSConstants.time_obj_col, CDMSConstants.year_col, CDMSConstants.month_col]
+ result = spark.sql(sql_stmt).coalesce(1).limit(self.__props.start_at + self.__props.size).drop(*removing_cols).tail(self.__props.size)
+ time_end = datetime.now()
+ LOGGER.debug(f'query_2 result duration: {time_end - time_start}')
+ return {'result': result}
+
+ def search(self, spark_session=None):
+ # LOGGER.debug(f'self.__sql_query(spark_session): {self.__sql_query(spark_session)}')
+ conditions = self.__add_conditions()
+ query_begin_time = datetime.now()
+ LOGGER.debug(f'query begins at {query_begin_time}')
+ spark = self.__retrieve_spark() if spark_session is None else spark_session
+ created_spark_session_time = datetime.now()
+ LOGGER.debug(f'spark session created at {created_spark_session_time}. duration: {created_spark_session_time - query_begin_time}')
+ read_df: DataFrame = spark.read.parquet(self.__parquet_name)
+ read_df_time = datetime.now()
+ LOGGER.debug(f'parquet read created at {read_df_time}. duration: {read_df_time - created_spark_session_time}')
+ query_result = read_df.where(conditions)
+ query_result = query_result.coalesce(1)
+ query_time = datetime.now()
+ LOGGER.debug(f'parquet read filtered at {query_time}. duration: {query_time - read_df_time}')
+ LOGGER.debug(f'total duration: {query_time - query_begin_time}')
+ total_result = int(query_result.coalesce(3).count())
+ # total_result = 1000 # faking this for now. TODO revert it.
+ LOGGER.debug(f'total calc count duration: {datetime.now() - query_time}')
+ if self.__props.size < 1:
+ LOGGER.debug(f'returning only the size: {total_result}')
+ return {
+ 'total': total_result,
+ 'results': [],
+ }
+ query_time = datetime.now()
+ # result = query_result.withColumn('_id', F.monotonically_increasing_id())
+ removing_cols = [CDMSConstants.time_obj_col, CDMSConstants.year_col, CDMSConstants.month_col]
+ # result = result.where(F.col('_id').between(self.__props.start_at, self.__props.start_at + self.__props.size)).drop(*removing_cols)
+ if len(self.__props.columns) > 0:
+ result = query_result.select(self.__props.columns)
+ LOGGER.debug(f'returning size : {total_result}')
+ result = query_result.limit(self.__props.start_at + self.__props.size).drop(*removing_cols).tail(self.__props.size)
+ LOGGER.debug(f'total retrieval duration: {datetime.now() - query_time}')
+ spark.stop()
+ return {
+ 'total': total_result,
+ 'results': [k.asDict() for k in result],
+ }
diff --git a/parquet_flask/io_logic/replace_file.py b/parquet_flask/io_logic/replace_file.py
new file mode 100644
index 0000000..669d229
--- /dev/null
+++ b/parquet_flask/io_logic/replace_file.py
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.io_logic.ingest_new_file import IngestNewJsonFile
+from parquet_flask.io_logic.retrieve_spark_session import RetrieveSparkSession
+from parquet_flask.io_logic.sanitize_record import SanitizeRecord
+from parquet_flask.utils.config import Config
+from parquet_flask.utils.file_utils import FileUtils
+
+from parquet_flask.utils.time_utils import TimeUtils
+
+LOGGER = logging.getLogger(__name__)
+
+
+class ReplaceJsonFile:
+ def __init__(self):
+ self.__sss = RetrieveSparkSession()
+ config = Config()
+ self.__app_name = config.get_value('spark_app_name')
+ self.__master_spark = config.get_value('master_spark_url')
+ self.__mode = 'overwrite'
+ self.__parquet_name = config.get_value('parquet_file_name')
+
+ def ingest(self, abs_file_path, job_id):
+ """
+ This method will assume that incoming file has data with in_situ_schema file.
+
+ So, it will definitely has `time`, `project`, and `provider`.
+
+ :param abs_file_path:
+ :param job_id:
+ :return:
+ """
+ if not FileUtils.file_exist(abs_file_path):
+ raise ValueError('missing file to ingest it. path: {}'.format(abs_file_path))
+ LOGGER.debug(f'sanitizing the files')
+ input_json = SanitizeRecord(Config().get_value('in_situ_schema')).start(abs_file_path)
+ spark_session = self.__sss.retrieve_spark_session(self.__app_name, self.__master_spark)
+ spark_session.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
+ df_writer = IngestNewJsonFile.create_df(spark_session,
+ input_json[CDMSConstants.observations_key],
+ job_id,
+ input_json[CDMSConstants.provider_col],
+ input_json[CDMSConstants.project_col])
+ df_writer.mode(self.__mode).parquet(self.__parquet_name, compression='GZIP')
+ LOGGER.debug(f'finished writing parquet')
+ return len(input_json[CDMSConstants.observations_key])
diff --git a/parquet_flask/io_logic/retrieve_spark_session.py b/parquet_flask/io_logic/retrieve_spark_session.py
new file mode 100644
index 0000000..2d6f185
--- /dev/null
+++ b/parquet_flask/io_logic/retrieve_spark_session.py
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pyspark import SparkConf
+from pyspark.sql import SparkSession
+
+from parquet_flask.utils.config import Config
+from parquet_flask.utils.singleton import Singleton
+
+
+class RetrieveSparkSession(metaclass=Singleton):
+ def __init__(self):
+ self.__sparks = {}
+
+ def retrieve_spark_session(self, app_name, master_spark, ram='3072m'):
+ session_key = '{}__{}'.format(app_name, master_spark)
+ # if session_key not in self.__sparks:
+ conf = SparkConf()
+ """
+ spark.executor.memory 3072m
+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3.S3FileSystem
+spark.hadoop.fs.s3n.impl org.apache.hadoop.fs.s3native.NativeS3FileSystem
+spark.driver.extraClassPath /usr/bin/spark-3.0.0-bin-hadoop3.2/jars/hadoop-aws-3.2.0.jar:/usr/bin/spark-3.0.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.11.563.jar
+spark.executor.extraClassPath /usr/bin/spark-3.0.0-bin-hadoop3.2/jars/hadoop-aws-3.2.0.jar:/usr/bin/spark-3.0.0-bin-hadoop3.2/jars/aws-java-sdk-bundle-1.11.563.jar
+spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true
+spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4=true
+ """
+ conf.set('spark.executor.memory', ram)
+ # conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.2.0') # crosscheck the version.
+ conf.set('spark.hadoop.fs.s3a.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem')
+ # conf.set('spark.driver.extraClassPath', '/opt/bitnami/spark/jars/hadoop-aws-3.2.0.jar:/opt/bitnami/spark/jars/aws-java-sdk-bundle-1.11.375.jar')
+ # conf.set('spark.executor.extraClassPath', '/opt/bitnami/spark/jars/hadoop-aws-3.2.0.jar:/opt/bitnami/spark/jars/aws-java-sdk-bundle-1.11.375.jar')
+ # conf.set('spark.executor.extraJavaOptions', '-Dcom.amazonaws.services.s3.enableV4=true')
+ # conf.set('spark.driver.extraJavaOptions', '-Dcom.amazonaws.services.s3.enableV4=true')
+ conf.set('spark.hadoop.fs.s3a.access.key', Config().get_value('aws_access_key_id'))
+ conf.set('spark.hadoop.fs.s3a.secret.key', Config().get_value('aws_secret_access_key'))
+ conf.set('spark.hadoop.fs.s3a.session.token', Config().get_value('aws_session_token'))
+ conf.set('spark.hadoop.fs.s3a.connection.ssl.enabled', 'true')
+ # conf.set('spark.default.parallelism', '10')
+ # conf.set('spark.hadoop.fs.s3a.endpoint', 's3.us-gov-west-1.amazonaws.com')
+ return SparkSession.builder.appName(app_name).config(conf=conf).master(master_spark).getOrCreate()
+ # self.__sparks[session_key] = SparkSession.builder.appName(app_name).config(conf=conf).master(master_spark).getOrCreate()
+ # return self.__sparks[session_key]
+
+ def stop_spark_session(self, app_name, master_spark):
+ session_key = '{}__{}'.format(app_name, master_spark)
+ if session_key in self.__sparks:
+ self.__sparks.pop(session_key).stop()
+ return
diff --git a/parquet_flask/io_logic/sanitize_record.py b/parquet_flask/io_logic/sanitize_record.py
new file mode 100644
index 0000000..962591f
--- /dev/null
+++ b/parquet_flask/io_logic/sanitize_record.py
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.utils.file_utils import FileUtils
+from parquet_flask.utils.general_utils import GeneralUtils
+from parquet_flask.utils.parallel_json_validator import ParallelJsonValidator
+
+LOGGER = logging.getLogger(__name__)
+
+
+basic_schema = {
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "title": "Cloud-based Data Match-Up Service In Situ Schema",
+ "description": "Schema for in situ data",
+ "properties": {
+ "provider": {
+ "description": "",
+ "type": "string"
+ },
+ "project": {
+ "description": "",
+ "type": "string"
+ },
+ "observations": {
+ "type": "array",
+ "items": {
+ "type": "object"
+ },
+ "minItems": 1
+ }
+ },
+ "required": [
+ "provider",
+ "project",
+ "observations",
+ ]
+}
+
+
+class SanitizeRecord:
+ def __init__(self, json_schema_path):
+ self.__json_schema_path = json_schema_path
+ if not FileUtils.file_exist(json_schema_path):
+ raise ValueError('json_schema file does not exist: {}'.format(json_schema_path))
+ self.__json_schema = FileUtils.read_json(json_schema_path)
+ self.__schema_key_values = {k: v for k, v in self.__json_schema['definitions']['observation']['properties'].items()}
+ self.__parallel_json_validator = ParallelJsonValidator()
+
+ def __sanitize_record(self, data_blk):
+ for k, v in data_blk.items():
+ if k in self.__schema_key_values and \
+ 'type' in self.__schema_key_values[k] and \
+ self.__schema_key_values[k]['type'] == 'number':
+ data_blk[k] = float(v)
+ return
+
+ def __validate_json(self, data):
+ LOGGER.debug(f'validating input data')
+ chunked_data = [{
+ "provider": data['provider'],
+ "project": data['project'],
+ 'observations': eachChunk,
+ } for eachChunk in GeneralUtils.chunk_list(data['observations'], 1000)]
+ if not self.__parallel_json_validator.is_schema_loaded():
+ self.__parallel_json_validator.load_schema(self.__json_schema)
+ result, error = self.__parallel_json_validator.validate_json(chunked_data)
+ return result, error
+
+ def start(self, json_file_path):
+ if not FileUtils.file_exist(json_file_path):
+ raise ValueError('json file does not exist: {}'.format(json_file_path))
+ json_obj = FileUtils.read_json(json_file_path)
+ is_valid, json_errors = GeneralUtils.is_json_valid(json_obj, basic_schema)
+ if not is_valid:
+ raise ValueError(f'input file has invalid high level schema: {json_file_path}. errors; {json_errors}')
+ LOGGER.warning('disabling validation of individual observation record. it is taking a long time')
+ is_valid, json_errors = self.__validate_json(json_obj)
+ if not is_valid:
+ raise ValueError(f'json has some error. Not validating: {json_errors}')
+ for each in json_obj[CDMSConstants.observations_key]:
+ self.__sanitize_record(each)
+ return json_obj
diff --git a/parquet_flask/utils/__init__.py b/parquet_flask/utils/__init__.py
new file mode 100644
index 0000000..fa8ccd5
--- /dev/null
+++ b/parquet_flask/utils/__init__.py
@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
\ No newline at end of file
diff --git a/parquet_flask/utils/config.py b/parquet_flask/utils/config.py
new file mode 100644
index 0000000..5861baf
--- /dev/null
+++ b/parquet_flask/utils/config.py
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from parquet_flask.utils.singleton import Singleton
+
+
+class Config(metaclass=Singleton):
+ def __init__(self):
+ self.__keys = [
+ 'master_spark_url',
+ 'spark_app_name',
+ 'parquet_file_name',
+ 'aws_access_key_id',
+ 'aws_secret_access_key',
+ 'aws_session_token',
+ 'in_situ_schema',
+ ]
+ self.__optional_keys = [
+ 'spark_ram_size',
+ ]
+ self.__validate()
+
+ def __validate(self):
+ missing_mandatory_keys = [k for k in self.__keys if k not in os.environ]
+ if len(missing_mandatory_keys) > 0:
+ raise RuntimeError('missing configuration values in environment values: {}'.format(missing_mandatory_keys))
+ return
+
+ def get_value(self, key, default_val=None):
+ if key in os.environ:
+ return os.environ[key]
+ return default_val
diff --git a/parquet_flask/utils/file_utils.py b/parquet_flask/utils/file_utils.py
new file mode 100644
index 0000000..d2622b7
--- /dev/null
+++ b/parquet_flask/utils/file_utils.py
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import hashlib
+import json
+import os
+import zlib
+from functools import partial
+from pathlib import Path
+from subprocess import Popen, PIPE
+
+
+class FileUtils:
+ @staticmethod
+ def mk_dir_p(dir_path):
+ Path(dir_path).mkdir(parents=True, exist_ok=True)
+ return
+
+ @staticmethod
+ def gunzip_file_os(zipped_file_path, output_file_path=None):
+ if not FileUtils.file_exist(zipped_file_path):
+ raise ValueError('missing file: {}'.format(zipped_file_path))
+ session = Popen(['gunzip', zipped_file_path], stdout=PIPE, stderr=PIPE)
+ stdout, stderr = session.communicate()
+ if stderr:
+ raise RuntimeError('error while gunzipping the file with Popen. filename: {}. error: {}'.format(zipped_file_path, stderr))
+ default_output_path = zipped_file_path[:-3]
+ if not FileUtils.file_exist(default_output_path):
+ raise ValueError('missing gunzipped file: {}'.format(default_output_path))
+ if output_file_path is None:
+ output_file_path = default_output_path
+ if FileUtils.file_exist(output_file_path) and default_output_path != output_file_path:
+ os.renames(default_output_path, output_file_path)
+ return output_file_path
+
+ @staticmethod
+ def get_checksum(file_path):
+ with open(file_path, mode='rb') as f:
+ d = hashlib.sha512()
+ for buf in iter(partial(f.read, 512 * 2**10), b''):
+ d.update(buf)
+ return d.hexdigest()
+
+ @staticmethod
+ def get_size(file_path):
+ return os.stat(file_path).st_size
+
+ @staticmethod
+ def file_exist(path):
+ return Path(path).is_file()
+
+ @staticmethod
+ def dir_exist(path):
+ return Path(path).is_dir()
+
+ @staticmethod
+ def del_file(path):
+ if FileUtils.file_exist(path):
+ Path(path).unlink()
+ return
+
+ @staticmethod
+ def read_json(path):
+ with open(path, 'r') as ff:
+ try:
+ return json.loads(ff.read())
+ except:
+ return None
+
+ @staticmethod
+ def write_json(file_path, json_obj, overwrite=False, append=False, prettify=False):
+ if os.path.exists(file_path) and not overwrite:
+ raise ValueError('{} already exists, and not overwriting'.format(file_path))
+ with open(file_path, 'a' if append else 'w') as ff:
+ json_str = json.dumps(json_obj, indent=4) if prettify else json.dumps(json_obj)
+ ff.write(json_str)
+ pass
+ return
diff --git a/parquet_flask/utils/general_utils.py b/parquet_flask/utils/general_utils.py
new file mode 100644
index 0000000..f1c2341
--- /dev/null
+++ b/parquet_flask/utils/general_utils.py
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import fastjsonschema
+
+
+class GeneralUtils:
+ @staticmethod
+ def is_json_valid(payload, schema):
+ try:
+ fastjsonschema.validate(schema, payload)
+ except Exception as error:
+ return False, str(error)
+ return True, None
+
+ @staticmethod
+ def chunk_list(input_list, chunked_size):
+ """Yield successive n-sized chunks from l."""
+ for i in range(0, len(input_list), chunked_size):
+ yield input_list[i:i + chunked_size]
diff --git a/parquet_flask/utils/parallel_json_validator.py b/parquet_flask/utils/parallel_json_validator.py
new file mode 100644
index 0000000..f769a26
--- /dev/null
+++ b/parquet_flask/utils/parallel_json_validator.py
@@ -0,0 +1,82 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import fastjsonschema
+import logging
+from datetime import datetime
+from multiprocessing import Pool
+from parquet_flask.utils.singleton import Singleton
+
+LOGGER = logging.getLogger(__name__)
+
+
+def __validate_small_data(small_data):
+ validator = ParallelJsonValidator()
+ try:
+ validator.schema(small_data)
+ return None
+ except Exception as e:
+ return str(e)
+
+
+def parallel_validate(chunked_data):
+ a = datetime.now()
+ with Pool(16) as p:
+ all_result = p.map(__validate_small_data, chunked_data)
+ all_result = [k for k in all_result if k is not None]
+ b = datetime.now()
+ LOGGER.debug(f'validation took: {b - a}')
+ return len(all_result) < 1, all_result
+
+
+class ParallelJsonValidator(metaclass=Singleton):
+ def __init__(self):
+ self.__schema = None
+
+ @property
+ def schema(self):
+ return self.__schema
+
+ @schema.setter
+ def schema(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__schema = fastjsonschema.compile(val)
+ return
+
+ def load_schema(self, input_schema):
+ self.schema = input_schema
+ return self
+
+ def is_schema_loaded(self):
+ return self.__schema is not None
+
+ def __validate_this(self, small_data):
+ try:
+ self.__schema(small_data)
+ return None
+ except Exception as e:
+ return str(e)
+
+ def validate_json(self, chunked_data: list):
+ if self.is_schema_loaded() is False:
+ raise ValueError(f'schema is not loaded. cannot validate')
+ if len(chunked_data) < 1:
+ LOGGER.debug(f'no need to validate empty json')
+ return True
+ LOGGER.debug(f'chunked_data size: {len(chunked_data)}')
+ return parallel_validate(chunked_data)
diff --git a/parquet_flask/utils/singleton.py b/parquet_flask/utils/singleton.py
new file mode 100644
index 0000000..d9d006a
--- /dev/null
+++ b/parquet_flask/utils/singleton.py
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class Singleton(type):
+ _instances = {}
+
+ def __call__(cls, *args, **kwargs):
+ if cls not in cls._instances:
+ cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
+ return cls._instances[cls]
+ pass
\ No newline at end of file
diff --git a/parquet_flask/utils/time_utils.py b/parquet_flask/utils/time_utils.py
new file mode 100644
index 0000000..407dca5
--- /dev/null
+++ b/parquet_flask/utils/time_utils.py
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from datetime import datetime
+
+
+class TimeUtils:
+ @staticmethod
+ def get_current_time_str():
+ return datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ')
+
+ @staticmethod
+ def get_current_time_unix():
+ return int(datetime.utcnow().timestamp() * 1000)
+
+ @staticmethod
+ def get_datetime_obj(dt_str, fmt='%Y-%m-%dT%H:%M:%SZ'):
+ return datetime.strptime(dt_str, fmt)
+
+ @staticmethod
+ def get_time_str(unix_timestamp, fmt='%Y-%m-%dT%H:%M:%SZ', in_ms=True):
+ converting_timestamp = unix_timestamp / 1000 if in_ms is True else unix_timestamp
+ return datetime.utcfromtimestamp(converting_timestamp).strftime(fmt)
diff --git a/parquet_flask/v1/__init__.py b/parquet_flask/v1/__init__.py
new file mode 100644
index 0000000..1af6707
--- /dev/null
+++ b/parquet_flask/v1/__init__.py
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from flask import Blueprint
+from flask_restx import Api
+from .apidocs import api as apidocs
+from .ingest_json_s3 import api as ingest_parquet_json_s3
+from .replace_json_s3 import api as replace_parquet_json_s3
+from .query_data import api as query_data
+from .query_data_doms import api as query_data_doms
+_version = "1.0"
+
+blueprint = Blueprint('parquet_flask', __name__, url_prefix='/{}'.format(_version))
+blueprint.register_blueprint(apidocs)
+
+api = Api(blueprint,
+ title='Parquet ingestion & query',
+ version=_version,
+ description='API to support the Parquet ingestion & query data',
+ doc='/doc/'
+ )
+
+# Register namespaces
+api.add_namespace(ingest_parquet_json_s3)
+api.add_namespace(replace_parquet_json_s3)
+api.add_namespace(query_data)
+api.add_namespace(query_data_doms)
diff --git a/parquet_flask/v1/apidocs.py b/parquet_flask/v1/apidocs.py
new file mode 100644
index 0000000..6372c31
--- /dev/null
+++ b/parquet_flask/v1/apidocs.py
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+
+from flask_restx import Resource, Namespace
+from flask import Blueprint, redirect, request, send_from_directory
+
+apidocs_path = Path(__file__).parent.joinpath('apidocs').resolve()
+
+api = Blueprint('apidocs', __name__, apidocs_path, '', url_prefix='/apidocs')
+
+@api.get('')
+def get_redirect():
+ # Appends a / to the requested path
+ return redirect(f'{request.path}/', 301)
+
+@api.get('/')
+def get_index():
+ return send_from_directory(apidocs_path, 'index.html')
diff --git a/parquet_flask/v1/apidocs/index.html b/parquet_flask/v1/apidocs/index.html
new file mode 100644
index 0000000..e2ea726
--- /dev/null
+++ b/parquet_flask/v1/apidocs/index.html
@@ -0,0 +1,61 @@
+<!-- HTML for static distribution bundle build -->
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="UTF-8">
+ <title>Swagger UI</title>
+ <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/swagger-ui/3.52.0/swagger-ui.min.css" integrity="sha512-OzAczs3EC865hvaGNsO241q+Mec0OEJ/DqGB33M1nt284+cAa8EPq/k1hUkQQxM+y0cN085INGU1WcZ12hoqfg==" crossorigin="anonymous" referrerpolicy="no-referrer" />
+ <style>
+ html
+ {
+ box-sizing: border-box;
+ overflow: -moz-scrollbars-vertical;
+ overflow-y: scroll;
+ }
+
+ *,
+ *:before,
+ *:after
+ {
+ box-sizing: inherit;
+ }
+
+ body
+ {
+ margin:0;
+ background: #fafafa;
+ }
+ </style>
+ </head>
+
+ <body>
+ <div id="swagger-ui"></div>
+
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/swagger-ui/3.52.0/swagger-ui-bundle.min.js" integrity="sha512-5Om3SqdVpHoZ1gYajhWvokmdj4Qr3YnMtCkgxOxraWo5tU4L40tyMHG+ZDmOS8AB+oEtF/0mA3FymbkYCOdZ7Q==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/swagger-ui/3.52.0/swagger-ui-standalone-preset.min.js" integrity="sha512-Pa5TWsES4T1o6gHBRqi9I5+quIPpDBjThcQLO7rvvaNYquPvAntd0Gtm7nOWGMQKY5ds6PeqccVBC6KAHqN+fQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
+ <script>
+ window.onload = function() {
+ // Begin Swagger UI call region
+ const ui = SwaggerUIBundle({
+ url: "insitu-spec-0.0.1.yml",
+ dom_id: '#swagger-ui',
+ deepLinking: true,
+ presets: [
+ SwaggerUIBundle.presets.apis,
+ SwaggerUIStandalonePreset
+ ],
+ plugins: [
+ SwaggerUIBundle.plugins.DownloadUrl
+ ],
+ syntaxHighlight: {
+ activated: false
+ },
+ layout: "StandaloneLayout"
+ });
+ // End Swagger UI call region
+
+ window.ui = ui;
+ };
+ </script>
+ </body>
+</html>
diff --git a/parquet_flask/v1/apidocs/insitu-spec-0.0.1.yml b/parquet_flask/v1/apidocs/insitu-spec-0.0.1.yml
new file mode 100644
index 0000000..dcec4ff
--- /dev/null
+++ b/parquet_flask/v1/apidocs/insitu-spec-0.0.1.yml
@@ -0,0 +1,263 @@
+openapi: 3.0.3
+info:
+ title: in-situ-data-services
+ description: API for querying in-situ data sources
+ version: 0.0.1
+ license:
+ name: Apache 2.0
+ url: http://www.apache.org/licenses/LICENSE-2.0.html
+
+servers:
+ - url: http://rda-data.ucar.edu:8890/ws/search
+ - url: http://doms.coaps.fsu.edu/ws/search
+ - url: https://doms.jpl.nasa.gov
+ - url: /
+ description: Current server
+
+paths:
+ '/{source}':
+ get:
+ summary: Query data source
+ tags:
+ - Query
+ parameters:
+ - in: path
+ name: source
+ required: true
+ description: |
+ Currently avalible sources:
+ - rda-data.ucar.edu:8890 - icoads
+ - doms.coaps.fsu.edu - samos
+ - doms.jpl.nasa.gov - spurs, spurs2
+ example: icoads
+ schema:
+ type: string
+ - $ref: '#/components/parameters/startTimeParam'
+ - $ref: '#/components/parameters/endTimeParam'
+ - $ref: '#/components/parameters/bboxParam'
+ - $ref: '#/components/parameters/variableParam'
+ - $ref: '#/components/parameters/minDepthParam'
+ - $ref: '#/components/parameters/maxDepthParam'
+ - $ref: '#/components/parameters/platformParam'
+ - $ref: '#/components/parameters/startIndexParam'
+ - $ref: '#/components/parameters/itemsPerPageParam'
+ - $ref: '#/components/parameters/prettyParam'
+ responses:
+ '200':
+ description: 'Successful query'
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/QueryResponse'
+ default:
+ description: 'Invalid query'
+
+components:
+ schemas:
+ QueryResponse:
+ type: object
+ properties:
+ last:
+ type: string
+ example: 'http://rda-data.ucar.edu:8890/ws/search/icoads?startIndex=928610&endTime=2013-10-31T23%3A59%3A59Z&bbox=-45%2C15%2C-30%2C30&startTime=2012-08-01T00%3A00%3A00Z'
+ next:
+ type: string
+ example: 'http://rda-data.ucar.edu:8890/ws/search/icoads?startIndex=10&endTime=2013-10-31T23%3A59%3A59Z&bbox=-45%2C15%2C-30%2C30&startTime=2012-08-01T00%3A00%3A00Z'
+ first:
+ type: string
+ example: 'http://rda-data.ucar.edu:8890/ws/search/icoads?startIndex=0&endTime=2013-10-31T23%3A59%3A59Z&bbox=-45%2C15%2C-30%2C30&startTime=2012-08-01T00%3A00%3A00Z'
+ results:
+ type: array
+ items:
+ $ref: '#/components/schemas/Result'
+ totalResults:
+ type: integer
+ example: 928618
+ minimum: 0
+ startIndex:
+ type: integer
+ example: 0
+ minimum: 0
+ itemsPerPage:
+ type: integer
+ example: 10
+ minimum: 0
+
+ Result:
+ type: object
+ properties:
+ id:
+ type: string
+ example: 'A1V2GP'
+ time:
+ type: string
+ format: date-time
+ example: '2012-09-01T07:00:00Z'
+ point:
+ type: string
+ pattern: 'Point\(-?\d{1,2}\.\d{1,2} -?\d{1,2}\.\d{1,2}\)'
+ example: 'Point(-35.90 25.70)'
+ sea_water_temperature:
+ type: number
+ example: 26.9
+ sea_water_temperature_depth:
+ type: number
+ nullable: true
+ sea_water_temperature_quality:
+ type: integer
+ example: 1
+ wind_speed:
+ type: number
+ example: 8.2
+ eastward_wind:
+ type: number
+ example: -7.1
+ northward_wind:
+ type: number
+ example: -4.1
+ wind_depth:
+ type: number
+ nullable: true
+ wind_speed_quality:
+ type: integer
+ example: 1
+ wind_component_quality:
+ type: integer
+ example: 1
+ sea_water_salinity:
+ type: number
+ nullable: true
+ sea_water_salinity_depth:
+ type: number
+ nullable: true
+ sea_water_salinity_quality:
+ type: integer
+ example: 9
+ mission:
+ type: integer
+ example: 2
+ platform:
+ type: integer
+ example: 1
+ device:
+ type: integer
+ nullable: true
+ metadata:
+ type: object
+ properties:
+ uid:
+ type: string
+ example: 'A1V2GP'
+ rn:
+ type: integer
+ example: 300
+ dck:
+ type: integer
+ example: 992
+ sid:
+ type: integer
+ example: 114
+ pt:
+ type: integer
+ example: 5
+ si:
+ type: integer
+ example: 3
+ wi:
+ type: integer
+ example: 4
+ di:
+ type: integer
+ example: 0
+ b10:
+ type: integer
+ example: 246
+ nd:
+ type: integer
+ example: 1
+ trms:
+ type: string
+ example: '111111'
+ nqcs:
+ type: string
+ example: '11AAA11A118AA1'
+
+ parameters:
+ startTimeParam:
+ in: query
+ name: startTime
+ description: 'Start time in the format of YYYY-MM-DDTHH:mm:ssZ'
+ example: '2012-08-01T00:00:00Z'
+ schema:
+ type: string
+ format: date-time
+ endTimeParam:
+ in: query
+ name: endTime
+ description: 'End time in the format of YYYY-MM-DDTHH:mm:ssZ'
+ example: '2013-10-31T23:59:59Z'
+ schema:
+ type: string
+ format: date-time
+ bboxParam:
+ in: query
+ name: bbox
+ description: 'Bounding box with coordinates specified in this order: west, south, east, north'
+ example: -45,15,-30,30
+ schema:
+ type: string
+ pattern: '-?\d{1,3},-?\d{1,2},-?\d{1,3},-?\d{1,2}'
+ variableParam:
+ in: query
+ name: variable
+ schema:
+ type: string
+ example: sss
+ enum:
+ - sss
+ - sst
+ - wind
+ minDepthParam:
+ in: query
+ name: minDepth
+ description: 'Minimum depth value in meters'
+ example: -5
+ schema:
+ type: integer
+ maxDepthParam:
+ in: query
+ name: maxDepth
+ description: 'Maximum depth value in meters'
+ example: 5
+ schema:
+ type: integer
+ platformParam:
+ in: query
+ name: platform
+ description: 'Platform ID'
+ example: 1
+ schema:
+ type: integer
+ startIndexParam:
+ in: query
+ name: startIndex
+ description: 'Start index of entries in the result; used for pagination'
+ example: 0
+ schema:
+ type: integer
+ default: 0
+ itemsPerPageParam:
+ in: query
+ name: itemsPerPage
+ description: 'Number of results per page'
+ example: 10
+ schema:
+ default: 10
+ type: integer
+ prettyParam:
+ in: query
+ name: pretty
+ description: 'Return formatted results if set to true'
+ example: true
+ schema:
+ type: boolean
diff --git a/parquet_flask/v1/ingest_aws_json.py b/parquet_flask/v1/ingest_aws_json.py
new file mode 100644
index 0000000..e003677
--- /dev/null
+++ b/parquet_flask/v1/ingest_aws_json.py
@@ -0,0 +1,156 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import uuid
+
+from parquet_flask.aws.aws_s3 import AwsS3
+from parquet_flask.io_logic.cdms_constants import CDMSConstants
+from parquet_flask.io_logic.ingest_new_file import IngestNewJsonFile
+from parquet_flask.io_logic.metadata_tbl_io import MetadataTblIO
+from parquet_flask.utils.file_utils import FileUtils
+from parquet_flask.utils.time_utils import TimeUtils
+
+LOGGER = logging.getLogger(__name__)
+
+
+class IngestAwsJsonProps:
+ def __init__(self):
+ self.__s3_url = None
+ self.__uuid = str(uuid.uuid4())
+ self.__working_dir = f'/tmp/{str(uuid.uuid4())}'
+ self.__is_replacing = False
+
+ @property
+ def is_replacing(self):
+ return self.__is_replacing
+
+ @is_replacing.setter
+ def is_replacing(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__is_replacing = val
+ return
+
+ @property
+ def working_dir(self):
+ return self.__working_dir
+
+ @working_dir.setter
+ def working_dir(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__working_dir = val
+ return
+
+ @property
+ def s3_url(self):
+ return self.__s3_url
+
+ @s3_url.setter
+ def s3_url(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__s3_url = val
+ return
+
+ @property
+ def uuid(self):
+ return self.__uuid
+
+ @uuid.setter
+ def uuid(self, val):
+ """
+ :param val:
+ :return: None
+ """
+ self.__uuid = val
+ return
+
+
+class IngestAwsJson:
+ def __init__(self, props=IngestAwsJsonProps()):
+ self.__props = props
+ self.__saved_file_name = None
+ self.__ingested_date = TimeUtils.get_current_time_unix()
+ self.__db_io = MetadataTblIO()
+
+ def ingest(self):
+ """
+ - download s3 file
+ - unzip if needed
+ - ingest to parquet
+ - update to metadata tbl
+ - delete local file
+ - tag s3 object
+
+ :return: tuple - (json object, return code)
+ """
+ try:
+ LOGGER.debug(f'starting to ingest: {self.__props.s3_url}')
+ existing_record = self.__db_io.get_by_s3_url(self.__props.s3_url)
+ if existing_record is None and self.__props.is_replacing is True:
+ LOGGER.error(f'unable to replace file as it is new. {self.__props.s3_url}')
+ return {'message': 'unable to replace file as it is new'}, 500
+
+ if existing_record is not None and self.__props.is_replacing is False:
+ LOGGER.error(f'unable to ingest file as it is already ingested. {self.__props.s3_url}. ingested record: {existing_record}')
+ return {'message': 'unable to ingest file as it is already ingested'}, 500
+
+ s3 = AwsS3().set_s3_url(self.__props.s3_url)
+ LOGGER.debug(f'downloading s3 file: {self.__props.uuid}')
+ FileUtils.mk_dir_p(self.__props.working_dir)
+ self.__saved_file_name = s3.download(self.__props.working_dir)
+ if self.__saved_file_name.lower().endswith('.gz'):
+ LOGGER.debug(f's3 file is in gzipped form. unzipping. {self.__saved_file_name}')
+ self.__saved_file_name = FileUtils.gunzip_file_os(self.__saved_file_name)
+ LOGGER.debug(f'ingesting file: {self.__saved_file_name}')
+ start_time = TimeUtils.get_current_time_unix()
+ num_records = IngestNewJsonFile().ingest(self.__saved_file_name, self.__props.uuid)
+ end_time = TimeUtils.get_current_time_unix()
+ LOGGER.debug(f'uploading to metadata table')
+ new_record = {
+ CDMSConstants.s3_url_key: self.__props.s3_url,
+ CDMSConstants.uuid_key: self.__props.uuid,
+ CDMSConstants.ingested_date_key: self.__ingested_date,
+ CDMSConstants.file_size_key: FileUtils.get_size(self.__saved_file_name),
+ CDMSConstants.checksum_key: FileUtils.get_checksum(self.__saved_file_name),
+ CDMSConstants.job_start_key: start_time,
+ CDMSConstants.job_end_key: end_time,
+ CDMSConstants.records_count_key: num_records,
+ }
+ if self.__props.is_replacing:
+ self.__db_io.replace_record(new_record)
+ else:
+ self.__db_io.insert_record(new_record)
+ LOGGER.debug(f'deleting used file')
+ FileUtils.del_file(self.__saved_file_name)
+ # TODO make it background process?
+ LOGGER.debug(f'tagging s3')
+ s3.add_tags_to_obj({
+ 'parquet_ingested': TimeUtils.get_time_str(self.__ingested_date),
+ 'job_id': self.__props.uuid,
+ })
+ return {'message': 'ingested'}, 201
+ except Exception as e:
+ LOGGER.debug(f'deleting error file')
+ FileUtils.del_file(self.__saved_file_name)
+ return {'message': 'failed to ingest to parquet', 'details': str(e)}, 500
diff --git a/parquet_flask/v1/ingest_json_s3.py b/parquet_flask/v1/ingest_json_s3.py
new file mode 100644
index 0000000..5623705
--- /dev/null
+++ b/parquet_flask/v1/ingest_json_s3.py
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from flask_restx import Resource, Namespace, fields
+from flask import request
+
+from parquet_flask.utils.general_utils import GeneralUtils
+from parquet_flask.v1.ingest_aws_json import IngestAwsJsonProps, IngestAwsJson
+
+api = Namespace('ingest_json_s3', description="Ingesting JSON files")
+LOGGER = logging.getLogger(__name__)
+
+query_model = api.model('ingest_json_s3', {
+ 's3_url': fields.String(required=True, example='s3://<bucket>/<key>'),
+})
+
+_QUERY_SCHEMA = {
+ 'type': 'object',
+ 'properties': {
+ 's3_url': {'type': 'string'},
+ },
+ 'required': ['s3_url'],
+}
+
+
+@api.route('', methods=["put"])
+class IngestParquet(Resource):
+ def __init__(self, api=None, *args, **kwargs):
+ super().__init__(api, args, kwargs)
+
+ @api.expect(fields=query_model)
+ def put(self):
+ """
+ s3://ecsv-h5-data-v1/INDEX/GALILEO/filenames.txt.gz
+
+ :return:
+ """
+ payload = request.get_json()
+ is_valid, json_error = GeneralUtils.is_json_valid(payload, _QUERY_SCHEMA)
+ if not is_valid:
+ return {'message': 'invalid request body', 'details': str(json_error)}, 400
+ props = IngestAwsJsonProps()
+ props.s3_url = payload["s3_url"]
+ return IngestAwsJson(props).ingest()
diff --git a/parquet_flask/v1/query_data.py b/parquet_flask/v1/query_data.py
new file mode 100644
index 0000000..74ac883
--- /dev/null
+++ b/parquet_flask/v1/query_data.py
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import logging
+
+from flask_restx import Resource, Namespace, fields
+from flask import request
+
+from parquet_flask.io_logic.query_v2 import QueryProps, Query, QUERY_PROPS_SCHEMA
+from parquet_flask.utils.general_utils import GeneralUtils
+
+api = Namespace('query_data', description="Querying data")
+LOGGER = logging.getLogger(__name__)
+
+query_model = api.model('query_data', {
+ 'start_from': fields.Integer(required=True, example=0),
+ 'size': fields.Integer(required=True, example=0),
+ 'provider': fields.String(required=True, example='JPL'),
+ 'project': fields.String(required=True, example='ABCD'),
+ 'min_depth': fields.Float(required=True, example=-65.34),
+ 'max_depth': fields.Float(required=True, example=-65.34),
+ 'min_time': fields.String(required=True, example='2020-01-01T00:00:00Z'),
+ 'max_time': fields.String(required=True, example='2020-01-31T00:00:00Z'),
+ 'columns': fields.List(fields.String, required=False, example=['latitudes', 'longitudes']),
+ 'min_lat_lon': fields.List(fields.Float, required=True, example=[-45, 175]),
+ 'max_lat_lon': fields.List(fields.Float, required=True, example=[-42.11, 175.16439819335938]),
+})
+
+
+@api.route('', methods=["get", "post"])
+class IngestParquet(Resource):
+ def __init__(self, api=None, *args, **kwargs):
+ super().__init__(api, args, kwargs)
+ self.__saved_dir = '/tmp' # TODO update this
+
+ def __execute_query(self, payload):
+ is_valid, json_error = GeneralUtils.is_json_valid(payload, QUERY_PROPS_SCHEMA)
+ if not is_valid:
+ return {'message': 'invalid request body', 'details': str(json_error)}, 400
+ try:
+ query = Query(QueryProps().from_json(payload))
+ result_set = query.search()
+ LOGGER.debug(f'search params: {payload}. result: {result_set}')
+ return {'result_set': result_set}, 200
+ except Exception as e:
+ LOGGER.exception(f'failed to query parquet. cause: {str(e)}')
+ return {'message': 'failed to query parquet', 'details': str(e)}, 500
+
+ @api.expect()
+ def get(self):
+ query_json = {
+ 'start_from': request.args.get('start_from', '0'),
+ 'size': request.args.get('size', '10'),
+ }
+ if 'min_time' in request.args:
+ query_json['min_time'] = request.args.get('min_time')
+ if 'max_time' in request.args:
+ query_json['max_time'] = request.args.get('max_time')
+ if 'min_depth' in request.args:
+ query_json['min_depth'] = float(request.args.get('min_depth'))
+ if 'max_depth' in request.args:
+ query_json['max_depth'] = float(request.args.get('max_depth'))
+ if 'min_lat_lon' in request.args:
+ query_json['min_lat_lon'] = json.loads(request.args.get('min_lat_lon'))
+ if 'max_lat_lon' in request.args:
+ query_json['max_lat_lon'] = json.loads(request.args.get('max_lat_lon'))
+ return self.__execute_query(query_json)
+
+ @api.expect()
+ def post(self):
+ """
+ s3://ecsv-h5-data-v1/INDEX/GALILEO/filenames.txt.gz
+
+ :return:
+ """
+ return self.__execute_query(request.get_json())
diff --git a/parquet_flask/v1/query_data_doms.py b/parquet_flask/v1/query_data_doms.py
new file mode 100644
index 0000000..7927195
--- /dev/null
+++ b/parquet_flask/v1/query_data_doms.py
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import logging
+from copy import deepcopy
+
+from flask_restx import Resource, Namespace, fields
+from flask import request
+
+from parquet_flask.io_logic.query_v2 import QueryProps, Query, QUERY_PROPS_SCHEMA
+from parquet_flask.utils.general_utils import GeneralUtils
+
+api = Namespace('query_data_doms', description="Querying data")
+LOGGER = logging.getLogger(__name__)
+
+query_model = api.model('query_data_doms', {
+ 'startIndex': fields.Integer(required=True, example=0),
+ 'itemsPerPage': fields.Integer(required=True, example=0),
+ 'minDepth': fields.Float(required=True, example=-65.34),
+ 'maxDepth': fields.Float(required=True, example=-65.34),
+ 'startTime': fields.String(required=True, example='2020-01-01T00:00:00Z'),
+ 'endTime': fields.String(required=True, example='2020-01-31T00:00:00Z'),
+ 'platform': fields.Integer(required=True, example=0),
+ 'provider': fields.Integer(required=True, example=0),
+ 'project': fields.Integer(required=True, example=0),
+ 'columns': fields.List(fields.String, required=False, example=['latitudes', 'longitudes']),
+ 'bbox': fields.List(fields.Float, required=True, example=[-45, 175, -30, 180]), # west, south, east, north
+})
+
+
+@api.route('', methods=["get", "post"])
+class IngestParquet(Resource):
+ def __init__(self, api=None, *args, **kwargs):
+ super().__init__(api, args, kwargs)
+ self.__start_from = 0
+ self.__size = 0
+
+ def __calculate_4_ranges(self, total_result):
+ if self.__size == 0:
+ return {
+ 'first': 0,
+ 'last': 0,
+ 'prev': 0,
+ 'next': 0,
+ }
+ div, mod = divmod(total_result, self.__size)
+ if mod > 0:
+ div += 1
+ page_info = {
+ 'first': 0,
+ 'last': div - 1,
+ 'prev': 0 if self.__start_from == 0 else self.__start_from - 1,
+ }
+ page_info['next'] = page_info['last'] if self.__start_from == page_info['last'] else self.__start_from + 1
+ return page_info
+
+ def __replace_start_from(self, new_start_from):
+ new_args = deepcopy(dict(request.args))
+ new_args['startIndex'] = new_start_from
+ return '&'.join([f'{k}={v}' for k, v in new_args.items()])
+
+ def __execute_query(self, payload):
+ """
+ TODO: transform the results to:
+ {
+ "last": "url",
+ "prev": "url",
+ "next": "url",
+ "first": "url",
+ "results": ["results"],
+ "total": "number
+ }
+ :param payload:
+ :return:
+ """
+ is_valid, json_error = GeneralUtils.is_json_valid(payload, QUERY_PROPS_SCHEMA)
+ if not is_valid:
+ return {'message': 'invalid request body', 'details': str(json_error)}, 400
+ try:
+ query = Query(QueryProps().from_json(payload))
+ result_set = query.search()
+ LOGGER.debug(f'search params: {payload}. result: {result_set}')
+ page_info = self.__calculate_4_ranges(result_set['total'])
+ result_set['last'] = f'{request.base_url}?{self.__replace_start_from(page_info["last"])}'
+ result_set['first'] = f'{request.base_url}?{self.__replace_start_from(page_info["first"])}'
+ result_set['next'] = f'{request.base_url}?{self.__replace_start_from(page_info["next"])}'
+ result_set['prev'] = f'{request.base_url}?{self.__replace_start_from(page_info["prev"])}'
+ return result_set, 200
+ except Exception as e:
+ LOGGER.exception(f'failed to query parquet. cause: {str(e)}')
+ return {'message': 'failed to query parquet', 'details': str(e)}, 500
+
+ @api.expect()
+ def get(self):
+ self.__start_from = int(request.args.get('startIndex', '0'))
+ self.__size = int(request.args.get('itemsPerPage', '10'))
+ query_json = {
+ 'start_from': self.__start_from,
+ 'size': self.__size,
+ }
+ if 'startTime' in request.args:
+ query_json['min_time'] = request.args.get('startTime')
+ if 'endTime' in request.args:
+ query_json['max_time'] = request.args.get('endTime')
+ if 'minDepth' in request.args:
+ query_json['min_depth'] = float(request.args.get('minDepth'))
+ if 'maxDepth' in request.args:
+ query_json['max_depth'] = float(request.args.get('maxDepth'))
+ if 'bbox' in request.args:
+ bounding_box = json.loads(request.args.get('bbox'))
+ query_json['min_lat_lon'] = [bounding_box[1], bounding_box[0]]
+ query_json['max_lat_lon'] = [bounding_box[3], bounding_box[2]]
+ if 'platform' in request.args:
+ query_json['platform_code'] = request.args.get('platform')
+ if 'provider' in request.args:
+ query_json['provider'] = request.args.get('provider')
+ if 'project' in request.args:
+ query_json['project'] = request.args.get('project')
+ return self.__execute_query(query_json)
diff --git a/parquet_flask/v1/replace_json_s3.py b/parquet_flask/v1/replace_json_s3.py
new file mode 100644
index 0000000..f51314b
--- /dev/null
+++ b/parquet_flask/v1/replace_json_s3.py
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from flask_restx import Resource, Namespace, fields
+from flask import request
+
+from parquet_flask.utils.general_utils import GeneralUtils
+from parquet_flask.v1.ingest_aws_json import IngestAwsJsonProps, IngestAwsJson
+
+api = Namespace('replace_json_s3', description="Ingesting JSON files")
+LOGGER = logging.getLogger(__name__)
+
+query_model = api.model('replace_json_s3', {
+ 's3_url': fields.String(required=True, example='s3://<bucket>/<key>'),
+ 'job_id': fields.String(required=True, example='sample-uuid'),
+})
+
+_QUERY_SCHEMA = {
+ 'type': 'object',
+ 'properties': {
+ 's3_url': {'type': 'string'},
+ 'job_id': {'type': 'string'},
+ },
+ 'required': ['s3_url', 'job_id'],
+}
+
+
+@api.route('', methods=["put"])
+class IngestParquet(Resource):
+ def __init__(self, api=None, *args, **kwargs):
+ super().__init__(api, args, kwargs)
+
+ @api.expect(fields=query_model)
+ def put(self):
+ """
+ s3://ecsv-h5-data-v1/INDEX/GALILEO/filenames.txt.gz
+
+ :return:
+ """
+ payload = request.get_json()
+ is_valid, json_error = GeneralUtils.is_json_valid(payload, _QUERY_SCHEMA)
+ if not is_valid:
+ return {'message': 'invalid request body', 'details': str(json_error)}, 400
+ props = IngestAwsJsonProps()
+ props.s3_url = payload['s3_url']
+ props.uuid = payload['job_id']
+ props.is_replacing = True
+ return IngestAwsJson(props).ingest()
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..a1960a0
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from setuptools import find_packages, setup
+
+# install_requires = [
+# 'pyspark===3.1.1',
+# # 'fastparquet===0.5.0', # not using it. sticking to pyspark with spark cluster according to Nga
+# 'findspark===1.4.2',
+# 'flask===1.1.2', 'flask_restful===0.3.8', 'flask-restx===0.3.0', # to create Flask server
+# 'gevent===1.4.0', 'greenlet===0.4.16', # to run flask server
+# 'werkzeug===0.16.1',
+# 'jsonschema', # to verify json objects
+# 'fastjsonschema===2.15.1',
+# 'boto3', 'botocore',
+# ]
+
+install_requires = [
+ 'pyspark===3.1.2',
+ # 'fastparquet===0.5.0', # not using it. sticking to pyspark with spark cluster according to Nga
+ 'findspark===1.4.2',
+ 'flask===2.0.1', 'flask_restful===0.3.9', 'flask-restx===0.5.0', # to create Flask server
+ 'gevent===21.8.0', 'greenlet===1.1.1', # to run flask server
+ 'werkzeug===2.0.1',
+ 'jsonschema', # to verify json objects
+ 'fastjsonschema===2.15.1',
+ 'boto3', 'botocore',
+]
+
+setup(
+ name="parquet_ingestion_search",
+ version="0.0.1",
+ packages=find_packages(),
+ install_requires=install_requires,
+ author="Apache SDAP",
+ author_email="dev@sdap.apache.org",
+ python_requires="==3.7",
+ license='NONE',
+ include_package_data=True,
+)