You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2022/12/01 20:53:10 UTC
[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #220: cdmssubset bugfix
skorper opened a new pull request, #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220
- Removed int requirement from list of platforms
- Made parameter optional
- Updated openapi spec
- Fixed issue in subsetter when multiple platforms are matched. Since this matches with different science variables, it's possible not every science variable will have a value for each row.
Tested with:
```bash
/cdmssubset?insitu=ICOADS Release 3.0&startTime=2018-10-21T00:00:00Z&endTime=2018-10-22T01:00:00Z&b=-30,15,-45,30&depthMin=0&depthMax=5&platforms=0,16,17,30,41,42&output=ZIP
```
Result looks as expected. [download.zip](https://github.com/apache/incubator-sdap-nexus/files/10136100/download.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #220: cdmssubset bugfix
Posted by GitBox <gi...@apache.org>.
RKuttruff commented on code in PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040163330
##########
analysis/webservice/algorithms/doms/subsetter.py:
##########
@@ -309,16 +300,21 @@ def toCsv(self):
]
data_variables = list(set([keys for result in results for keys in result['data'].keys()]))
data_variables.sort()
+
+ if 'id' in list(set([keys for result in results for keys in result.keys()])):
+ headers.append('id')
+
headers.extend(data_variables)
for i, result in enumerate(results):
cols = []
cols.append(result['longitude'])
cols.append(result['latitude'])
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+ cols.append(result.get('id'))
Review Comment:
Gotcha 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #220: cdmssubset bugfix
Posted by GitBox <gi...@apache.org>.
skorper commented on code in PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040162217
##########
analysis/webservice/algorithms/doms/subsetter.py:
##########
@@ -309,16 +300,21 @@ def toCsv(self):
]
data_variables = list(set([keys for result in results for keys in result['data'].keys()]))
data_variables.sort()
+
+ if 'id' in list(set([keys for result in results for keys in result.keys()])):
+ headers.append('id')
+
headers.extend(data_variables)
for i, result in enumerate(results):
cols = []
cols.append(result['longitude'])
cols.append(result['latitude'])
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+ cols.append(result.get('id'))
Review Comment:
Thanks Riley, I actually made that change locally but never pushed it! I split the id from the default header because there are some cases where "id" is not present and some where there aren't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #220: cdmssubset bugfix
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220#issuecomment-1335870590
Update: Added insitu uid to cdmssubset request as requested by Amanda. I just used the entire `meta` field for now, which includes the uid in the icoads case.
Example: [ICOADS Release 3.0.csv](https://github.com/apache/incubator-sdap-nexus/files/10144539/ICOADS.Release.3.0.csv)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper merged pull request #220: cdmssubset bugfix
Posted by "skorper (via GitHub)" <gi...@apache.org>.
skorper merged PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #220: cdmssubset bugfix
Posted by GitBox <gi...@apache.org>.
RKuttruff commented on code in PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040150796
##########
analysis/webservice/algorithms/doms/subsetter.py:
##########
@@ -309,16 +300,21 @@ def toCsv(self):
]
data_variables = list(set([keys for result in results for keys in result['data'].keys()]))
data_variables.sort()
+
+ if 'id' in list(set([keys for result in results for keys in result.keys()])):
+ headers.append('id')
+
headers.extend(data_variables)
for i, result in enumerate(results):
cols = []
cols.append(result['longitude'])
cols.append(result['latitude'])
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+ cols.append(result.get('id'))
Review Comment:
I think this should only be called if the id header was added. Otherwise I'm seeing csv output with 4 header fields and 5 data fields which is being incorrectly parsed by pandas
Eg:
```
longitude,latitude,time,sea_surface_foundation_temperature
172.625,-29.875,2018-09-29T09:00:00Z,None,19.509003
...
```
Parsed as:
```
>>> csv_data
longitude latitude time sea_surface_foundation_temperature
172.625 -29.875 2018-09-29T09:00:00Z None 19.509003
172.875 -29.875 2018-09-29T09:00:00Z None 18.703003
173.125 -29.875 2018-09-29T09:00:00Z None 18.308990
173.375 -29.875 2018-09-29T09:00:00Z None 18.298004
173.625 -29.875 2018-09-29T09:00:00Z None 18.329010
... ... ... ... ...
171.375 -25.125 2018-09-24T09:00:00Z None 21.669006
171.625 -25.125 2018-09-24T09:00:00Z None 21.360992
171.875 -25.125 2018-09-24T09:00:00Z None 21.075012
172.125 -25.125 2018-09-24T09:00:00Z None 20.730011
172.375 -25.125 2018-09-24T09:00:00Z None 20.737000
[9600 rows x 4 columns]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-nexus] skorper commented on pull request #220: cdmssubset bugfix
Posted by GitBox <gi...@apache.org>.
skorper commented on PR #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220#issuecomment-1338813448
Using the latest commit, generated the zip below with the following request: `/cdmssubset?dataset=ASCATB-L2-Coastal&insitu=ICOADS Release 3.0&startTime=2017-07-01T00:00:00Z&endTime=2017-07-01T23:59:59Z&b=-90,27,-85,29&depthMin=0&depthMax=5&platforms=0,16,17,30,41,42&output=ZIP`
[response.zip](https://github.com/apache/incubator-sdap-nexus/files/10163119/response.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sdap.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org