You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2022/04/13 01:53:39 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #158: SDAP-372: Integrate AWS insitu API
skorper opened a new pull request, #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158
In this PR, I've integrated the AWS Parquet insitu API to the matchup algorithm.
Please be aware that the insitu API is very slow as of now, so we may want to hold off on merging/deploying this until those issues have been resolved. For example, the following request:
```
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&tt=2592000&rt=1000&b=160,-30,180,-25&depthMin=0&depthMax=5&matchOnce=true&secondary=ICOADS%20Release%203.0&resultSizeLimit=6000&platforms=42
```
takes over 4 minutes to complete. Granted, this is on my local machine which is only using a single Cassandra node and a single Spark node. However this is still significantly slower than previous requests using the legacy insitu API.
In addition to the insitu API integration, this PR also includes a new `MatchupException`. If raised, the message will be passed directly to users. I'm hoping this will be the start of better and more meaningful error messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1144320844
@ngachung Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung merged pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
ngachung merged PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] ngachung commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
ngachung commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1144304337
Let's go with a single top level changelog. Not sure that component level changelog will get enough attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1136492386
Tested with the insitu API now that the parallel issue has been resolved -- it's working now. The following request:
```
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&tt=2592000&rt=1000&b=160,-30,180,-25&depthMin=0&depthMax=5&matchOnce=true&secondary=ICOADS%20Release%203.0&resultSizeLimit=6000&platforms=42
```
Takes 2m 15.48s when running from my local machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1124171764
@skorper This is currently blocked by an issue with 504 errors when rapidly hitting the CDMS insitu API. @wphyojpl is looking into this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] wphyojpl commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
wphyojpl commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1100243672
Hi @skorper
So, I created this URL from the example in the description:
```
https://doms.jpl.nasa.gov/insitu/1.0/query_data_doms_custom_pagination?startIndex=0&itemsPerPage=20000&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&bbox=160,-30,180,-25&minDepth=0.0&maxDepth=5.0&provider=NCAR&project=ICOADS%20Release%203.0&platform=42
```
I mainly changed the time range and bbox.
This is what I get:
```
(base) MT-212437:~ wphyo$ time curl 'https://doms.jpl.nasa.gov/insitu/1.0/query_data_doms_custom_pagination?startIndex=0&itemsPerPage=20000&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&bbox=160,-30,180,-25&minDepth=0.0&maxDepth=5.0&provider=NCAR&project=ICOADS%20Release%203.0&platform=42'
{"total": 283, "results": [{"depth": -99999.0, "latitude": -28.38, "longitude": 160.47, "meta": "https://rda.ucar.edu/php/icoadsuid.php?uid=P9IL5A", "platform": {"type": "7", "code": "42", "id": "55960"}, "time": "2018-09-24T00:00:00Z", "provider": "NCAR", "proj
.
.
.
}
real 0m9.196s
user 0m0.020s
sys 0m0.021s
```
So, it took around 5 to 9 second when I run it repeatedly..
But it only has 283 items.
It shouldn't take 4 min.. We can debug it together if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #158: SDAP-372: Integrate AWS insitu API
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158#issuecomment-1144299359
@ngachung I added two changelogs -- one for `data-access` and one for `analysis`. I'm not sure if we want to do it this way or do a single changelog for the entire repo due to the entire repo going into a single docker image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org