You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2022/09/28 00:19:56 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #202: SDAP-402: Update matchOnce logic
skorper opened a new pull request, #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202
[SDAP-402](https://issues.apache.org/jira/browse/SDAP-402)
- Fixed matchOnce logic. Previously, only a single secondary point was matched to a primary point. Now, we are allowing multiple matches so long as they share the same time and space as the closest point.
- Fixed the failing matchup unit tests
- Added a new unit test case for this change
- Fixed bug where `depth` was being overwritten with a null value (maybe will fix our depth problems?)
Tested this locally with the following request (from the google doc):
```
{{big_data_url}}/match_spark?tt=43200&primary=JPL-L4-MRVA-CHLA-GLOB-v3.0&secondary=shark-2018&startTime=2018-04-01T00%3A00%3A00Z&endTime=2018-04-01T12%3A59%3A59Z&rt=25000&b=-140%2C10%2C-110%2C40&platforms=3B&depthMin=-5&depthMax=5&matchOnce=true
```
<details>
<summary>This results in the following json output:</summary>
```json
{
"executionId": "e4fb1b4f-c002-4c68-a651-94db87e96c2f",
"data": [
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-129.875",
"lat": "27.125",
"point": "Point(-129.875 27.125)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 18, 20]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.0518328994512558,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-130.015296",
"lat": "27.0813824",
"point": "Point(-130.015296 27.0813824)",
"time": 1522570620,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.11,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.015296",
"lat": "27.0813824",
"point": "Point(-130.015296 27.0813824)",
"time": 1522570620,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.628,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 18.822,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-129.875",
"lat": "27.375",
"point": "Point(-129.875 27.375)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 19, 20]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05334627255797386,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "102",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": -2.3,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_temperature",
"cf_variable_name": "air_temperature",
"variable_value": 18.58,
"variable_unit": "C"
},
{
"variable_name": "air_temperature_quality",
"cf_variable_name": "air_temperature_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "relative_humidity",
"cf_variable_name": "relative_humidity",
"variable_value": 60.88,
"variable_unit": "1"
},
{
"variable_name": "relative_humidity_quality",
"cf_variable_name": "relative_humidity_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "102",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": -0.2,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_pressure",
"cf_variable_name": "air_pressure",
"variable_value": 1019.2,
"variable_unit": "hPa"
},
{
"variable_name": "air_pressure_quality",
"cf_variable_name": "air_pressure_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "113",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.09,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.669,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.623,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-130.125",
"lat": "27.375",
"point": "Point(-130.125 27.375)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 19, 19]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05735308676958084,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "102",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": -2.3,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_temperature",
"cf_variable_name": "air_temperature",
"variable_value": 18.58,
"variable_unit": "C"
},
{
"variable_name": "air_temperature_quality",
"cf_variable_name": "air_temperature_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "relative_humidity",
"cf_variable_name": "relative_humidity",
"variable_value": 60.88,
"variable_unit": "1"
},
{
"variable_name": "relative_humidity_quality",
"cf_variable_name": "relative_humidity_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "102",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": -0.2,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_pressure",
"cf_variable_name": "air_pressure",
"variable_value": 1019.2,
"variable_unit": "hPa"
},
{
"variable_name": "air_pressure_quality",
"cf_variable_name": "air_pressure_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "113",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.09,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.020352",
"lat": "27.240928",
"point": "Point(-130.020352 27.240928)",
"time": 1522540800,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.669,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.623,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-130.125",
"lat": "26.875",
"point": "Point(-130.125 26.875)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 17, 19]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05785427987575531,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-129.9747584",
"lat": "26.8175136",
"point": "Point(-129.9747584 26.8175136)",
"time": 1522608780,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.08,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-129.9747584",
"lat": "26.8175136",
"point": "Point(-129.9747584 26.8175136)",
"time": 1522608780,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.565,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 18.797,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-130.125",
"lat": "27.125",
"point": "Point(-130.125 27.125)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 18, 19]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05617373436689377,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-130.0269312",
"lat": "27.1240416",
"point": "Point(-130.0269312 27.1240416)",
"time": 1522564980,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.12,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.0269312",
"lat": "27.1240416",
"point": "Point(-130.0269312 27.1240416)",
"time": 1522564980,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.618,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 18.847,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-130.125",
"lat": "26.625",
"point": "Point(-130.125 26.625)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 16, 19]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05913911014795303,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "102",
"lon": "-130.0557312",
"lat": "26.6866656",
"point": "Point(-130.0557312 26.6866656)",
"time": 1522627200,
"depth": -2.3,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_temperature",
"cf_variable_name": "air_temperature",
"variable_value": 18.09,
"variable_unit": "C"
},
{
"variable_name": "air_temperature_quality",
"cf_variable_name": "air_temperature_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "relative_humidity",
"cf_variable_name": "relative_humidity",
"variable_value": 66.92,
"variable_unit": "1"
},
{
"variable_name": "relative_humidity_quality",
"cf_variable_name": "relative_humidity_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "102",
"lon": "-130.0557312",
"lat": "26.6866656",
"point": "Point(-130.0557312 26.6866656)",
"time": 1522627200,
"depth": -0.2,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "air_pressure",
"cf_variable_name": "air_pressure",
"variable_value": 1018.12,
"variable_unit": "hPa"
},
{
"variable_name": "air_pressure_quality",
"cf_variable_name": "air_pressure_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "113",
"lon": "-130.0557312",
"lat": "26.6866656",
"point": "Point(-130.0557312 26.6866656)",
"time": 1522627200,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.08,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.0557312",
"lat": "26.6866656",
"point": "Point(-130.0557312 26.6866656)",
"time": 1522627200,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.624,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.177,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-129.875",
"lat": "26.875",
"point": "Point(-129.875 26.875)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 17, 20]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.05762673169374466,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-129.9238912",
"lat": "26.8958944",
"point": "Point(-129.9238912 26.8958944)",
"time": 1522597380,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.12,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-129.9238912",
"lat": "26.8958944",
"point": "Point(-129.9238912 26.8958944)",
"time": 1522597380,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.677,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 18.909,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
},
{
"platform": "orbiting satellite",
"device": "radiometers",
"lon": "-129.875",
"lat": "26.625",
"point": "Point(-129.875 26.625)",
"time": 1522584000,
"depth": null,
"fileurl": "20180401120000-JPL-CHL25-fv03.nc",
"id": "96bc2f4b-fd78-3c41-a3cb-cc6b2d7197a3[[0, 16, 20]]",
"source": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"primary": [
{
"variable_name": "CHLA_analysis",
"cf_variable_name": "mass_concentration_of_chlorophyll_a_in_sea_water",
"variable_value": 0.061026107519865036,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-130.0324224",
"lat": "26.721784",
"point": "Point(-130.0324224 26.721784)",
"time": 1522621860,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.08,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.0324224",
"lat": "26.721784",
"point": "Point(-130.0324224 26.721784)",
"time": 1522621860,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.631,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.126,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
}
],
"params": {
"primary": "JPL-L4-MRVA-CHLA-GLOB-v3.0",
"matchup": "shark-2018",
"startTime": 1522540800,
"endTime": 1522587599,
"bbox": "-140,10,-110,40",
"timeTolerance": 43200,
"radiusTolerance": 25000.0,
"platforms": "3B",
"parameter": null,
"depthMin": -5.0,
"depthMax": 5.0
},
"bounds": {},
"count": 8,
"details": {
"timeToComplete": 119,
"numSecondaryMatched": 22,
"numPrimaryMatched": 8
}
}
```
</details>
Notice how there are 8 primary matches but 22 secondary matches. If we look at one of the matches, we can see the secondary points are at the same time/space as all the secondary matches.
When running the above query on the old algorithm we get the following counts:
`matchOnce=true`:
```json
{
"timeToComplete": 39,
"numSecondaryMatched": 8,
"numPrimaryMatched": 8
}
```
`matchOnce=false`:
```json
{
"timeToComplete": 32,
"numSecondaryMatched": 10784,
"numPrimaryMatched": 8
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1275745444
@ngachung The duplicate primary issue seems to occur prior to these changes, so it shouldn't block moving forward with this PR. All conflicts have been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
RKuttruff commented on PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1262998052
I actually don't know exactly how many matches L2 -> Saildrone _should_ be returning. I based the current test case on how many were returning from the AWS deployment. However, the current deployment is returning the 'expected' number of matches while the new changes have it returning 2 extra. One thing to note about these extra matches is that the primary points are actually duplicated in the matchup output.
An example for one of the extra matches:
<details>
<summary> Common point </summary>
```json
{
"platform": "orbiting satellite",
"device": "scatterometers",
"lon": "-129.94937",
"lat": "26.56576",
"point": "Point(-129.94937 26.56576)",
"time": 1522695015,
"depth": null,
"fileurl": "ascat_20180402_180900_metopb_28742_eps_o_coa_2401_ovw.l2.nc",
"id": "44d09fab-f85e-3de5-9495-835b8607d8e2[[9 9 9]]",
"source": "ASCATB-L2-Coastal",
"primary": [
{
"variable_name": "wind_speed",
"cf_variable_name": "wind_speed",
"variable_value": 2.4800000190734863,
"variable_unit": null
},
{
"variable_name": "wind_dir",
"cf_variable_name": "wind_to_direction",
"variable_value": 283.3999938964844,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "130",
"lon": "-130.0027776",
"lat": "26.35588",
"point": "Point(-130.0027776 26.35588)",
"time": 1522691460,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.681,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.314,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
}
```
</details>
<details>
<summary> Extra point (from modified SDAP) </summary>
```json
{
"platform": "orbiting satellite",
"device": "scatterometers",
"lon": "-129.94937",
"lat": "26.56576",
"point": "Point(-129.94937 26.56576)",
"time": 1522695015,
"depth": null,
"fileurl": "ascat_20180402_180900_metopb_28742_eps_o_coa_2401_ovw.l2.nc",
"id": "44d09fab-f85e-3de5-9495-835b8607d8e2[[9 9 9]]",
"source": "ASCATB-L2-Coastal",
"primary": [
{
"variable_name": "wind_speed",
"cf_variable_name": "wind_speed",
"variable_value": 2.4800000190734863,
"variable_unit": null
},
{
"variable_name": "wind_dir",
"cf_variable_name": "wind_to_direction",
"variable_value": 283.3999938964844,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-130.0174208",
"lat": "26.3396",
"point": "Point(-130.0174208 26.3396)",
"time": 1522694880,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water",
"variable_value": 0.08,
"variable_unit": "kg m-3"
},
{
"variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"cf_variable_name": "mass_concentration_of_chlorophyll_in_sea_water_quality",
"variable_value": 2,
"variable_unit": null
}
]
},
{
"platform": "3B",
"device": "130",
"lon": "-130.0174208",
"lat": "26.3396",
"point": "Point(-130.0174208 26.3396)",
"time": 1522694880,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": "sea_water_practical_salinity",
"cf_variable_name": "sea_water_practical_salinity",
"variable_value": 34.69,
"variable_unit": "1"
},
{
"variable_name": "sea_water_practical_salinity_quality",
"cf_variable_name": "sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": "sea_water_temperature",
"cf_variable_name": "sea_water_temperature",
"variable_value": 19.45,
"variable_unit": "C"
},
{
"variable_name": "sea_water_temperature_quality",
"cf_variable_name": "sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
}
```
</details>
Full outputs:
[outputs.zip](https://github.com/apache/incubator-sdap-nexus/files/9679943/outputs.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
RKuttruff commented on PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1261347211
As a followup: All the (currently implemented) test cases are passing with the exception of L2 -> Saildrone (returning more matches than expected)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
RKuttruff commented on PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1261314820
Testing to see if this affects my regression tests. It seems that the same query is returning different numbers of matches.
The query I'm using:
`http://localhost:8083/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-08-17T01%3A00%3A00Z&endTime=2018-08-27T06%3A00%3A00Z&tt=43200&rt=1000&b=-90.375,27.625,-86.125,28.125&depthMin=-20&depthMax=10&matchOnce=true&secondary=ICOADS%20Release%203.0&resultSizeLimit=7000&platforms=42`
I've seen it return 3, 4, and 5 matches for different runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1262983746
Note, L2 -> Saildrone is returning more matches than expected for both my changes and the version deployed to AWS (Riley correct me if I'm wrong) so this discrepancy is maybe unrelated to this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #202: SDAP-402: Update matchOnce logic
Posted by GitBox <gi...@apache.org>.
ngachung merged PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org