You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2021/04/03 00:34:45 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
skorper opened a new pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126
[Jira ticket](https://bugs.earthdata.nasa.gov/browse/CDMS-50)
Removed the `parameter` requirement on the matchup endpoint. If no parameter is provided, the behavior of `sst`, `sss` and `wind` will be combined.
Example:
```
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=1442102400&endTime=1442288800&tt=86400&rt=1000&b=-67.43499976462817,60.39585573909287,-50.681836983854154,76.72666987833477&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&matchup=icoads
```
Response:
```json
{
"executionId": "73d1bb41-f853-4574-b05a-20b7af26f130",
"data": [
{
"sea_water_temperature": 276.22998046875,
"sea_water_temperature_depth": 0,
"sea_water_salinity": 276.22998046875,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": 276.22998046875,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-62.875",
"y": "75.125",
"point": "Point(-62.875 75.125)",
"time": 1442102400,
"fileurl": "20150913120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "38e752cf-72bf-387e-8641-8f45d11b32ee[[0, 0, 18]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matches": [
{
"sea_water_temperature": 3.2,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "drifting surface float",
"device": null,
"x": "-62.88",
"y": "75.13",
"point": "Point(-62.88 75.13)",
"time": 1442124000,
"fileurl": null,
"id": "LCAHED",
"source": "icoads"
}
]
},
{
"sea_water_temperature": 276.1499938964844,
"sea_water_temperature_depth": 0,
"sea_water_salinity": 276.1499938964844,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": 276.1499938964844,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-62.875",
"y": "75.125",
"point": "Point(-62.875 75.125)",
"time": 1442188800,
"fileurl": "20150914120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "6ba8741b-4eb8-30f6-9470-3ae2b8fa5f63[[0, 0, 18]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matches": [
{
"sea_water_temperature": 3.2,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "drifting surface float",
"device": null,
"x": "-62.88",
"y": "75.13",
"point": "Point(-62.88 75.13)",
"time": 1442124000,
"fileurl": null,
"id": "LCAHED",
"source": "icoads"
}
]
}
],
"params": {
"primary": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matchup": "icoads",
"startTime": 1442102400,
"endTime": 1442288800,
"bbox": "-67.43499976462817,60.39585573909287,-50.681836983854154,76.72666987833477",
"timeTolerance": 86400,
"radiusTolerance": 1000.0,
"platforms": "1,2,3,4,5,6,7,8,9",
"parameter": null,
"depthMin": 0.0,
"depthMax": 5.0
},
"bounds": {},
"count": null,
"details": {
"timeToComplete": 27,
"numInSituRecords": 0,
"numInSituMatched": 2,
"numGriddedChecked": 0,
"numGriddedMatched": 2
}
}
```
Notice that the sst, sss, and wind values are provided in the response. Note: This is not deployed on bigdata yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on a change in pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper commented on a change in pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#discussion_r609998182
##########
File path: analysis/webservice/algorithms_spark/Matchup.py
##########
@@ -356,11 +356,11 @@ def from_nexus_point(nexus_point, tile=None, parameter='sst'):
point.data_id = "%s[%s]" % (tile.tile_id, nexus_point.index)
# TODO Not an ideal solution; but it works for now.
- if parameter == 'sst':
+ if parameter == 'sst' or parameter == None:
point.sst = nexus_point.data_val.item()
- elif parameter == 'sss':
+ if parameter == 'sss' or parameter == None:
point.sss = nexus_point.data_val.item()
- elif parameter == 'wind':
+ if parameter == 'wind' or parameter == None:
point.wind_u = nexus_point.data_val.item()
Review comment:
You're right, I hadn't noticed the values were the same for sst, sss, and wind. I'll continue working this ticket, thanks for your review @ngachung
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper edited a comment on pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper edited a comment on pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#issuecomment-820912622
I've updated this branch as-per our conversation so the satellite portion of the result will use the `tile_var_name_s` field as the data key. I left the other values in the response for consistency, but I think we should discuss if this is the desired behavior.
Example:
```
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-09-30T23:59:59Z&tt=86400&rt=1000&b=-45,15,-30,30&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&matchup=spurs
```
```json
{
"executionId": "0df70819-ee2a-4629-8fa8-23ccdad69e94",
"data": [
{
"sea_water_temperature": null,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-38.375",
"y": "26.125",
"point": "Point(-38.375 26.125)",
"time": 1348963200,
"fileurl": "20120930120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "26eb597e-c2f4-3b38-8439-d233593a03c6[[0, 14, 26]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"analysed_sst": 300.7599792480469,
"matches": [
{
"sea_water_temperature": 27.1054000854,
"sea_water_temperature_depth": 4.96795892715,
"sea_water_salinity": 37.6000518799,
"sea_water_salinity_depth": 4.96795892715,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "ship",
"device": "CTD",
"x": "-38.3727989197",
"y": "26.1182975769",
"point": "Point(-38.3727989197 26.1182975769)",
"time": 1349018581,
"fileurl": "ftp://podaac-ftp.jpl.nasa.gov/allData/insitu/L2/spurs1/uctd/SPURS_Knorr_uctd.nc",
"id": "SPURS_Knorr_uctd(4, 525)",
"source": "spurs"
}
]
}
],
"params": {
"primary": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matchup": "spurs",
"startTime": 1348531200,
"endTime": 1349049599,
"bbox": "-45,15,-30,30",
"timeTolerance": 86400,
"radiusTolerance": 1000.0,
"platforms": "1,2,3,4,5,6,7,8,9",
"parameter": null,
"depthMin": 0.0,
"depthMax": 5.0
},
"bounds": {},
"count": null,
"details": {
"timeToComplete": 47,
"numInSituRecords": 0,
"numInSituMatched": 1,
"numGriddedChecked": 0,
"numGriddedMatched": 1
}
}
```
Notice how the sss, sst, and wind fields in the satellite doms entry are empty, but there is a new field `analysed_sst` that contains the nexus value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#issuecomment-812765614
@ngachung
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
ngachung merged pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung commented on a change in pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
ngachung commented on a change in pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#discussion_r609992630
##########
File path: analysis/webservice/algorithms_spark/Matchup.py
##########
@@ -356,11 +356,11 @@ def from_nexus_point(nexus_point, tile=None, parameter='sst'):
point.data_id = "%s[%s]" % (tile.tile_id, nexus_point.index)
# TODO Not an ideal solution; but it works for now.
- if parameter == 'sst':
+ if parameter == 'sst' or parameter == None:
point.sst = nexus_point.data_val.item()
- elif parameter == 'sss':
+ if parameter == 'sss' or parameter == None:
point.sss = nexus_point.data_val.item()
- elif parameter == 'wind':
+ if parameter == 'wind' or parameter == None:
point.wind_u = nexus_point.data_val.item()
Review comment:
@skorper I don't think this change as is does what we need it to do. Taking a closer look at this, this algorithm makes the assumption that when invoking the API the user knows that the satellite dataset chosen stores either sst, sss, or wind variable. So with this change what happens is if user chooses a wind dataset such as ASCAT but doesn't restrict the parameter then all returned variables will have the same value.
The sample output you provided illustrates this issue:
```
"sea_water_temperature": 276.22998046875,
"sea_water_temperature_depth": 0,
"sea_water_salinity": 276.22998046875,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": 276.22998046875,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-62.875",
"y": "75.125",
"point": "Point(-62.875 75.125)",
"time": 1442102400,
"fileurl": "20150913120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
```
Note that the value 276.22998046875 is sst and not wind, sss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#issuecomment-820912622
I've updated this branch as-per our conversation so the satellite portion of the result will use the `tile_var_name_s` field as the data key. I left the other values in the response for consistency, but I think we should discuss if this is the desired behavior.
Example:
```
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2012-09-25T00:00:00Z&endTime=2012-09-30T23:59:59Z&tt=86400&rt=1000&b=-45,15,-30,30&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&matchup=spurs
```
```json
{
"executionId": "0df70819-ee2a-4629-8fa8-23ccdad69e94",
"data": [
{
"sea_water_temperature": null,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-38.375",
"y": "26.125",
"point": "Point(-38.375 26.125)",
"time": 1348963200,
"fileurl": "20120930120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "26eb597e-c2f4-3b38-8439-d233593a03c6[[0, 14, 26]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"analysed_sst": 300.7599792480469,
"matches": [
{
"sea_water_temperature": 27.1054000854,
"sea_water_temperature_depth": 4.96795892715,
"sea_water_salinity": 37.6000518799,
"sea_water_salinity_depth": 4.96795892715,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "ship",
"device": "CTD",
"x": "-38.3727989197",
"y": "26.1182975769",
"point": "Point(-38.3727989197 26.1182975769)",
"time": 1349018581,
"fileurl": "ftp://podaac-ftp.jpl.nasa.gov/allData/insitu/L2/spurs1/uctd/SPURS_Knorr_uctd.nc",
"id": "SPURS_Knorr_uctd(4, 525)",
"source": "spurs"
}
]
}
],
"params": {
"primary": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"matchup": "spurs",
"startTime": 1348531200,
"endTime": 1349049599,
"bbox": "-45,15,-30,30",
"timeTolerance": 86400,
"radiusTolerance": 1000.0,
"platforms": "1,2,3,4,5,6,7,8,9",
"parameter": null,
"depthMin": 0.0,
"depthMax": 5.0
},
"bounds": {},
"count": null,
"details": {
"timeToComplete": 47,
"numInSituRecords": 0,
"numInSituMatched": 1,
"numGriddedChecked": 0,
"numGriddedMatched": 1
}
}
```
Notice how the sss, sst, and wind fields in the satellite doms entry are empty, but there a new field `analysed_sst` that contains the nexus value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#issuecomment-820914466
Something to keep in mind here is that the provided `parameter` will never impact the nexus/tile search/results. The tiles are found only using dataset, temporal, and spatial bounds. Maybe we want to add in the ability to search for tiles by param?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper edited a comment on pull request #126: CDMS-50: Updated matchup algorithm to remove parameter restriction
Posted by GitBox <gi...@apache.org>.
skorper edited a comment on pull request #126:
URL: https://github.com/apache/incubator-sdap-nexus/pull/126#issuecomment-820914466
Something to keep in mind here is that the provided `parameter` will never impact the nexus/tile search results. The tiles are found only using dataset, temporal, and spatial bounds. Maybe we want to add in the ability to search for tiles by param?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org