You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2021/07/23 10:49:55 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
skorper opened a new pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131
Support satellite to satellite matchup in the matchup algorithm. This works as follows:
1. Satellite dataset name is used for `matchup` param rather than insitu dataset name.
2. Instead of querying the in situ node, query Solr to get the tiles that match the given params. This request is very similar to the first tile query request except the matchup params are used (time and spatial bounds are delta'd from the primary dataset tiles)
3. Once the tiles have been retrieved for the matchup satellite dataset, the individual points are extracted from each tile and flattened into a single list.
4. The points are converted to 'edge point' format. This matches the format of the edge points from the in-situ node.
5. Matchup continues as usual (reprojection, KD tree, etc)
Example:
```
{{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2018-10-30T00:00:00Z&endTime=2018-11-05T00:00:00Z&tt=2592000&rt=1000&b=-180,-90,-160,-70&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&matchup=MUR25-JPL-L4-GLOB-v04.2&resultSizeLimit=6000
```
(results truncated)
```json
{
"executionId": "1d821a81-896f-4f23-89bc-8c31d8b81b5d",
"data": [
{
"sea_water_temperature": null,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-161.625",
"y": "-74.875",
"point": "Point(-161.625 -74.875)",
"time": 1540857600,
"fileurl": "20181030120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "1be456ef-dffd-3efb-a967-491b9960661b[[0, 0, 13]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"sea_surface_temperature": 271.3800048828125,
"matches": [
{
"sea_water_temperature": null,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": null,
"device": null,
"x": "-161.625",
"y": "-74.875",
"point": "Point(-161.625 -74.875)",
"time": 1542099600,
"fileurl": "20181113090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc",
"id": "2018-11-13T09:00:00Z:-161.625:-74.875",
"source": "MUR25-JPL-L4-GLOB-v04.2",
"sea_surface_foundation_temperature": 271.3500061035156
}
]
},
{
"sea_water_temperature": null,
"sea_water_temperature_depth": 0,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": "orbiting satellite",
"device": "radiometers",
"x": "-163.625",
"y": "-74.625",
"point": "Point(-163.625 -74.625)",
"time": 1540857600,
"fileurl": "20181030120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc",
"id": "1be456ef-dffd-3efb-a967-491b9960661b[[0, 1, 5]]",
"source": "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",
"sea_surface_temperature": 271.3699951171875,
"matches": [
{
"sea_water_temperature": null,
"sea_water_temperature_depth": null,
"sea_water_salinity": null,
"sea_water_salinity_depth": null,
"wind_speed": null,
"wind_direction": null,
"wind_u": null,
"wind_v": null,
"platform": null,
"device": null,
"x": "-163.625",
"y": "-74.625",
"point": "Point(-163.625 -74.625)",
"time": 1542099600,
"fileurl": "20181113090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc",
"id": "2018-11-13T09:00:00Z:-163.625:-74.625",
"source": "MUR25-JPL-L4-GLOB-v04.2",
"sea_surface_foundation_temperature": 271.3500061035156
}
]
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131#issuecomment-889538765
@fgreg Thanks! I'm not sure what happened there, it looks like branch is really far behind so I'll try to bring it in and see how it goes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
ngachung merged pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
ngachung merged pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] fgreg commented on pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
fgreg commented on pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131#issuecomment-889193054
This commit on the 1.1.0-SNAPSHOT branch never made it to master
https://github.com/skorper/incubator-sdap-nexus/commit/c9de8ec9a9597f23f743efe1091d69943e800460#diff-826cfd666976053c90a1cf5b8de07143a88fe6090e5cb356d9cf8a941b2c878a
Those unit tests were checking the expected behavior of the tolerances and should be calling `webservice.algorithms_spark.Matchup.match_tile_to_point_generator`
I'm not sure where development is currently happening but there are commits on 1.1.0-SNAPSHOT that do not exist in master
https://github.com/apache/incubator-sdap-nexus/compare/1.1.0-SNAPSHOT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
skorper commented on pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131#issuecomment-888728581
@ngachung I've added some unit tests (matchup only for now) that exercise the satellite to satellite functionality. Changes are as follows:
- renamed matchup test module to `test_matchup.py`
- Removed tests that contained syntax error (no idea what they were supposed to be calling but it didn't exist)
- Moved integration tests to a new location and added integration pytest marker (I have not tested these existing integration tests)
- Added 2 new tests to test_matchup.py that exercise the bulk of the matchup algorithm
- Added a README to the test directory with instructions on how to run the unit tests.
Note: The exists tests are not in a functional state, the only tests I've worked on are the matchup tests.
```sh
➜ pytest analysis/tests/algorithms_spark/test_matchup.py
======================================================================================================================================================================================== test session starts =========================================================================================================================================================================================
platform darwin -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/skperez/Documents/ACCESS/incubator-sdap-nexus/analysis
collected 3 items
analysis/tests/algorithms_spark/test_matchup.py ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper edited a comment on pull request #131: SDAP-315: Updated matchup algorithm to support satellite to satellite
Posted by GitBox <gi...@apache.org>.
skorper edited a comment on pull request #131:
URL: https://github.com/apache/incubator-sdap-nexus/pull/131#issuecomment-888728581
@ngachung I've added some unit tests (matchup only for now) that exercise the satellite to satellite functionality. Changes are as follows:
- renamed matchup test module to `test_matchup.py`
- Removed tests that contained syntax error (no idea what they were supposed to be calling but it didn't exist)
- Moved integration tests to a new location and added integration pytest marker (I have not tested these existing integration tests)
- Added 2 new tests to test_matchup.py that exercise the bulk of the matchup algorithm
- For these tests, I've hardcoded the primary and secondary points and hand calculated the expected result.
- Added a README to the test directory with instructions on how to run the unit tests.
Note: The exists tests are not in a functional state, the only tests I've worked on are the matchup tests.
```sh
➜ pytest analysis/tests/algorithms_spark/test_matchup.py
======================================================================================================================================================================================== test session starts =========================================================================================================================================================================================
platform darwin -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/skperez/Documents/ACCESS/incubator-sdap-nexus/analysis
collected 3 items
analysis/tests/algorithms_spark/test_matchup.py ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org