You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by "skorper (via GitHub)" <gi...@apache.org> on 2023/09/25 22:57:21 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #282: SDAP-493: Pagination improvements
skorper opened a new pull request, #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282
- Removed `resultSizeLimit` from /match_spark. This is no longer needed as the result size limit is specified in the cdmsresults pagination endpoint.
- Updated /job output with two new fields:
- `totalPrimaryMatched`: The total number of primary matches
- `averageSecondaryMatched`: The average number of secondaries per primary. Calculated like so: `round(num_secondaries/num_primaries)`
- This will always be 1 when matchOnce=True
- Updated /job endpoint output from `jobId` to `executionId` to maintain consistency with `cdmsresults` endpoint.
---
### Example:
Query:
```
{{big_data_url}}/match_spark?primary=VIIRS_NPP-2018_Heatwave&secondary=ASCATB-L2-Coastal&startTime=2018-08-01T00%3A00%3A00Z&endTime=2018-08-01T23%3A59%3A59Z&b=-122%2C%2026%2C%20-114%2C%2034&platforms=30&depthMin=-1&depthMax=1&tt=43200&rt=18750&matchOnce=false&prioritizeDistance=true
```
Output:
```json
{
"status": "success",
"message": "",
"createdAt": "2023-09-25 22:35:27.598000",
"updatedAt": "2023-09-25 22:36:41.819000",
"links": [
{
"href": "http://localhost:8083/job?id=39bc409f-e990-4b38-b3a5-b1dfd41eae06",
"title": "The current page",
"type": "application/json",
"rel": "self"
},
{
"href": "http://localhost:8083/cdmsresults?id=39bc409f-e990-4b38-b3a5-b1dfd41eae06&output=CSV",
"title": "Download results",
"type": "text/csv",
"rel": "data"
},
{
"href": "http://localhost:8083/cdmsresults?id=39bc409f-e990-4b38-b3a5-b1dfd41eae06&output=JSON",
"title": "Download results",
"type": "application/json",
"rel": "data"
},
{
"href": "http://localhost:8083/cdmsresults?id=39bc409f-e990-4b38-b3a5-b1dfd41eae06&output=NETCDF",
"title": "Download results",
"type": "binary/octet-stream",
"rel": "data"
}
],
"params": {
"primary": "MUR25-JPL-L4-GLOB-v04.2",
"matchup": "ASCATB-L2-Coastal",
"startTime": "2018-08-01 00:00:00+00:00",
"endTime": "2018-08-31 23:59:59+00:00",
"bbox": "-122,26,-114,34",
"timeTolerance": 43200,
"radiusTolerance": 18750.0,
"platforms": "30",
"parameter": null,
"depthMin": -1.0,
"depthMax": 1.0
},
"executionID": "39bc409f-e990-4b38-b3a5-b1dfd41eae06",
"totalPrimaryMatched": 17475,
"averageSecondaryMatched": 8
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #282: SDAP-493: Pagination improvements
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper commented on code in PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#discussion_r1336467124
##########
analysis/webservice/webmodel/NexusExecutionResults.py:
##########
@@ -53,6 +54,9 @@ def construct_done(status, created, completed, execution_id, params, host):
params,
host
)
+ # Add stats to body
+ job_body['totalPrimaryMatched'] = num_primary_matched
+ job_body['averageSecondaryMatched'] = round(num_secondary_matched/num_primary_matched)
Review Comment:
I'm open to your idea of not rounding. I added rounding because I wasn't sure how valuable that extra info would be and I think it's nice that both values are integers representing a count. Maybe we can see if the team has a strong opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #282: SDAP-493: Pagination improvements
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on code in PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#discussion_r1337272582
##########
analysis/webservice/webmodel/NexusExecutionResults.py:
##########
@@ -53,6 +54,9 @@ def construct_done(status, created, completed, execution_id, params, host):
params,
host
)
+ # Add stats to body
+ job_body['totalPrimaryMatched'] = num_primary_matched
+ job_body['averageSecondaryMatched'] = round(num_secondary_matched/num_primary_matched)
Review Comment:
Couldn't hurt IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #282: SDAP-493: Pagination improvements
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on code in PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#discussion_r1336469307
##########
analysis/webservice/webmodel/NexusExecutionResults.py:
##########
@@ -53,6 +54,9 @@ def construct_done(status, created, completed, execution_id, params, host):
params,
host
)
+ # Add stats to body
+ job_body['totalPrimaryMatched'] = num_primary_matched
+ job_body['averageSecondaryMatched'] = round(num_secondary_matched/num_primary_matched)
Review Comment:
I think it may be worth bringing up. Ultimately, it's the total number of points that has the greatest affect on fetch time + size, so I feel an accurate way to determine that would be helpful for particularly large jobs / large tolerances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #282: SDAP-493: Pagination improvements
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper commented on PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#issuecomment-1741507534
@ngachung @RKuttruff I propose we add a new column to the `doms_execution_stats` table to store the number of unique secondary matches. IMO, this is the cleanest way to add this information that aligns with how we're already dealing with execution stats.
This is mostly completerd in commit dd73036307a313eaba57203f208196da2f6a3ab0, and I'll proceed to testing if everyone agrees with this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #282: SDAP-493: Pagination improvements
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on code in PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#discussion_r1336462012
##########
analysis/webservice/webmodel/NexusExecutionResults.py:
##########
@@ -53,6 +54,9 @@ def construct_done(status, created, completed, execution_id, params, host):
params,
host
)
+ # Add stats to body
+ job_body['totalPrimaryMatched'] = num_primary_matched
+ job_body['averageSecondaryMatched'] = round(num_secondary_matched/num_primary_matched)
Review Comment:
I'm curious why you decided to round to nearest int here. I think returning this as a float could benefit by allowing the user to compute the number of secondaries / total number of points more accurately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SDAP-493: Pagination improvements [incubator-sdap-nexus]
Posted by "ngachung (via GitHub)" <gi...@apache.org>.
ngachung merged PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #282: SDAP-493: Pagination improvements
Posted by "RKuttruff (via GitHub)" <gi...@apache.org>.
RKuttruff commented on PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#issuecomment-1741511545
> @ngachung @RKuttruff I propose we add a new column to the `doms_execution_stats` table to store the number of unique secondary matches. IMO, this is the cleanest way to add this information that aligns with how we're already dealing with execution stats.
>
> This is mostly completed in commit dd73036307a313eaba57203f208196da2f6a3ab0, and I'll proceed to testing if everyone agrees with this approach.
>
>
@skorper
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #282: SDAP-493: Pagination improvements
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper commented on code in PR #282:
URL: https://github.com/apache/incubator-sdap-nexus/pull/282#discussion_r1336470056
##########
analysis/webservice/webmodel/NexusExecutionResults.py:
##########
@@ -53,6 +54,9 @@ def construct_done(status, created, completed, execution_id, params, host):
params,
host
)
+ # Add stats to body
+ job_body['totalPrimaryMatched'] = num_primary_matched
+ job_body['averageSecondaryMatched'] = round(num_secondary_matched/num_primary_matched)
Review Comment:
I'm also curious if the users would find the raw count of secondaries useful. We could add that to the output directly if that's something they might find useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org