You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2021/03/25 20:38:58 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
skorper opened a new pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124
[Jira ticket](https://issues.apache.org/jira/browse/SDAP-305)
This branch adds the changes needed to get AVHRR OI matchup working on Kubernetes.
Changes:
- Added missing `tile_service_factory` param to `/domsresults`
- For doms endpoints, instead of just reading config directly and using those values, use the values passed in during runtime and overwrite. This is consistent with how the working non-doms endpoints are doing things.
- Updated pyspark to 3.1.1
- Updated kubernetes client to 4.12.0
This is deployed on bigdata. Example:
```
https://bigdata.jpl.nasa.gov/nexus/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=1442102400&endTime=1442288800&tt=86400&rt=1000&b=-67.43499976462817,60.39585573909287,-50.681836983854154,76.72666987833477&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&matchup=icoads
```
```json
{
"executionId": "25e88ec1-9d19-4f97-af5c-20019712d777",
"data": [
{
"sea_water_temperature": 276.22998046875,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-62.875",
"y": "75.125",
"point": "Point(-62.875 75.125)",
"time": 1442102400,
"fileurl": "20150913120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "38e752cf-72bf-387e-8641-8f45d11b32ee[[0, 0, 18]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matches": [
{
"sea_water_temperature": 3.2,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "drifting surface float",
"device": null,
"x": "-62.88",
"y": "75.13",
"point": "Point(-62.88 75.13)",
"time": 1442124000,
"fileurl": null,
"id": "LCAHED",
"source": "icoads"
}
]
},
{
"sea_water_temperature": 276.1499938964844,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-62.875",
"y": "75.125",
"point": "Point(-62.875 75.125)",
"time": 1442188800,
"fileurl": "20150914120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "6ba8741b-4eb8-30f6-9470-3ae2b8fa5f63[[0, 0, 18]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matches": [
{
"sea_water_temperature": 3.2,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "drifting surface float",
"device": null,
"x": "-62.88",
"y": "75.13",
"point": "Point(-62.88 75.13)",
"time": 1442124000,
"fileurl": null,
"id": "LCAHED",
"source": "icoads"
}
]
}
],
"params": {
"primary": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matchup": "icoads",
"startTime": 1442102400,
"endTime": 1442288800,
"bbox": "-67.43499976462817,60.39585573909287,-50.681836983854154,76.72666987833477",
"timeTolerance": 86400,
"radiusTolerance": 1000.0,
"platforms": "1,2,3,4,5,6,7,8,9",
"parameter": "sst",
"depthMin": 0.0,
"depthMax": 5.0
},
"bounds": {},
"count": null,
"details": {
"timeToComplete": 9,
"numInSituRecords": 0,
"numInSituMatched": 2,
"numGriddedChecked": 0,
"numGriddedMatched": 2
}
}
```
```
https://bigdata.jpl.nasa.gov/nexus/domsresults?id=da1f3ac6-79c8-452c-b899-d9c9816fbf92&simpleResults=true
```
```json
{
"executionId": "da1f3ac6-79c8-452c-b899-d9c9816fbf92",
"data": [
{
"x": -62.875,
"y": 75.125,
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"time": 1442188800,
"sea_water_temperature": 276.1499938964844,
"sea_water_temperature_depth": 0.0,
"matches": [
{
"x": -62.88,
"y": 75.13,
"source": "icoads",
"time": 1442124000,
"sea_water_temperature": 3.2
}
]
},
{
"x": -62.875,
"y": 75.125,
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"time": 1442102400,
"sea_water_temperature": 276.22998046875,
"sea_water_temperature_depth": 0.0,
"matches": [
{
"x": -62.88,
"y": 75.13,
"source": "icoads",
"time": 1442124000,
"sea_water_temperature": 3.2
}
]
}
],
"params": {
"primary": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matchup": [
"icoads"
],
"depthMin": "0.0",
"depthMax": "5.0",
"timeTolerance": 86400,
"radiusTolerance": "1000.0",
"startTime": 1442102400,
"endTime": 1442288800,
"platforms": "1,2,3,4,5,6,7,8,9",
"bbox": "-67.43499976462817,60.39585573909287,-50.681836983854154,76.72666987833477",
"parameter": "sst"
},
"bounds": {},
"count": null,
"details": {
"numGriddedMatched": 2,
"numGriddedChecked": 0,
"numInSituMatched": 2,
"numInSituChecked": 0,
"timeToComplete": 8
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124#issuecomment-808760303
@ngachung
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124#issuecomment-810678488
@ngachung I've committed a fix that was needed for the kubernetes deployment, which is deployed on bigdata.
The following requests are working as expected:
```
bigdata.jpl.nasa.gov/nexus/domsplot?id=33f27334-5a76-4e3c-83a7-15dd69bdf769&type=histogram
```
```
bigdata.jpl.nasa.gov/nexus/domsplot?id=33f27334-5a76-4e3c-83a7-15dd69bdf769&type=scatter
```
```
bigdata.jpl.nasa.gov/nexus/domsplot?id=33f27334-5a76-4e3c-83a7-15dd69bdf769&type=map
```
```
bigdata.jpl.nasa.gov/nexus/domsinsitusubset?source=spurs&startTime=2013-10-21T00:00:00Z&endTime=2013-10-31T23:59:59Z&b=-30,15,-45,30&depthMin=0&depthMax=5&output=JSON¶meter=sst
```
```
bigdata.jpl.nasa.gov/nexus/datainbounds?ds=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2018-10-21T00:00:00Z&endTime=2018-10-31T23:59:59Z&b=-20,-10,0,0¶meter=sst
```
```
bigdata.jpl.nasa.gov/nexus/domssubset?dataset=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020¶meter=sst&startTime=2013-10-21T00:00:00Z&endTime=2013-10-31T23:59:59Z&b=-30,15,-45,30&output=ZIP
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124#issuecomment-809836798
@ngachung The latest changes are working for me locally but need to be tested on bigdata once the SSL issues have been resolved (I didn't commit the `verify=False` workaround). For my local tests I changed the doms.jpl.nasa.gov request to localhost to get this to work as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
ngachung merged pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung commented on pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
ngachung commented on pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124#issuecomment-808982428
@skorper Thank you! Is it easy enough to get all the existing doms endpoints working? e.g. /domsplot, /domssubset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #124: SDAP-305: Get current match up working with AVHRR OI data that is currently ingested in the bigdata cluster
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #124:
URL: https://github.com/apache/incubator-sdap-nexus/pull/124#issuecomment-809536773
@ngachung I'll look into that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org