You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@climate.apache.org by Omkar20895 <gi...@git.apache.org> on 2016/08/08 19:38:05 UTC
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
GitHub user Omkar20895 opened a pull request:
https://github.com/apache/climate/pull/383
Adding PO.DAAC data source along with the tests
- This PR adds PO.DAAC data source to ocw.
- This PR also addresses the required tests.
Any suggestions are welcome, Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Omkar20895/climate CLIMATE-769
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/climate/pull/383.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #383
----
commit 116d16f039f2f3e3f1c4cfddec447a49afc459aa
Author: Omkar20895 <om...@gmail.com>
Date: 2016-08-08T19:36:40Z
Adding PO.DAAC data source along with the tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/climate/pull/383
[](https://coveralls.io/builds/7625197)
Coverage remained the same at 80.622% when pulling **7d0daf0c0accd9a60d64a773a31f31fa33cc8679 on Omkar20895:CLIMATE-769** into **91ec7f31c0c1cd6ac77372de7409f070f5d1e723 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@jarifibrahim what does this failure mean?
https://travis-ci.org/apache/climate/builds/151333565
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by agoodm <gi...@git.apache.org>.
Github user agoodm commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 I have already made the JIRA issue for it. I can quickly get this done next week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
Hi @Omkar20895 yes please. Go ahead and merge. We can work on Podaacpy 1.1.0 and then update here when necessary. Thanks very much. Excellent work and very well done over the summer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452030
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
+import numpy as np
+from ocw.dataset import Dataset
+from netCDF4 import Dataset as netcdf_dataset
+from netcdftime import utime
+import os
+import urllib
+import xml.etree.ElementTree as ET
+
+
+def convert_times_to_datetime(time):
+ '''Convert the time object's values to datetime objects
+
+ The time values are stored as some unit since an epoch. These need to be
+ converted into datetime objects for the OCW Dataset object.
+
+ :param time: The time object's values to convert
+ :type time: pydap.model.BaseType
+
+ :returns: list of converted time values as datetime objects
+ '''
+ units = time.units
+ # parse the time units string into a useful object.
+ # NOTE: This assumes a 'standard' calendar. It's possible (likely?) that
+ # users will want to customize this in the future.
+ parsed_time = utime(units)
+ return [parsed_time.num2date(x) for x in time[:]]
+
+
+def load_dataset(variable, datasetId='', datasetShortName='', name=''):
+ '''Loads a Dataset from PODAAC
+
+ :param variable: The name of the variable to read from the dataset.
+ :type variable: :mod:`string`
+
+ :param datasetId: dataset persistent ID. datasetId or \
+ shortName is required for a granule search. Example: \
+ PODAAC-ASOP2-25X01
+ :type datasetId: :mod:`string`
+
+ :param shortName: the shorter name for a dataset. \
+ Either shortName or datasetId is required for a \
+ granule search. Example: ASCATA-L2-25km
+ :type shortName: :mod:`string`
+
+ :param name: (Optional) A name for the loaded dataset.
+ :type name: :mod:`string`
+
+ :returns: A :class:`dataset.Dataset` containing the dataset pointed to by
+ the OpenDAP URL.
+
+ :raises: ServerError
+ '''
+ # Downloading the dataset using podaac toolkit
+ podaac = Podaac()
+ path = os.path.dirname(os.path.abspath(__file__))
+ granuleName = podaac.extract_l4_granule(
+ datasetId=datasetId, shortName=datasetShortName, path=path)
+ path = path + '/' + granuleName
+ d = netcdf_dataset(path, mode='r')
+ dataset = d.variables[variable]
+
+ # By convention, but not by standard, if the dimensions exist, they will be in the order:
--- End diff --
Make this Python function documentation rather than hidden docs please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc sure thing!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452491
--- Diff: examples/podaac_integration_example.py ---
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import ocw.data_source.podaac as podaac
+import ocw.evaluation as evaluation
+import ocw.metrics as metrics
+import ocw.plotter as plotter
+
+datasetId = 'PODAAC-CCF30-01XXX'
+datasetShortName = 'CCMP_MEASURES_ATLAS_L4_OW_L3_0_WIND_VECTORS_FLK'
+variable = 'uwnd'
+name = 'PO.DAAC_test_dataset'
+OUTPUT_PLOT = "cmc_temporal_std"
+""" Step 1: Load Local NetCDF Files into OCW Dataset Objects """
+print("Loading %s dataset into an OCW dataset objects" % datasetShortName)
+cmc_dataset = podaac.load_dataset(
+ variable=variable, datasetId=datasetId, datasetShortName=datasetShortName, name=name)
+print "CMC_Dataset.values shape: (times, lats, lons) - %s \n" % (cmc_dataset.values.shape,)
+
+# Acessing latittudes and longitudes of netCDF file
+lats = cmc_dataset.lats
+lons = cmc_dataset.lons
+
+""" Step 2: Build a Metric to use for Evaluation - Temporal STD for this example """
+# You can build your own metrics, but OCW also ships with some common metrics
+print "Setting up a Temporal STD metric to use for evaluation"
+std = metrics.TemporalStdDev()
+
+""" Step 3: Create an Evaluation Object using Datasets and our Metric """
+# The Evaluation Class Signature is:
+# Evaluation(reference, targets, metrics, subregions=None)
+# Evaluation can take in multiple targets and metrics, so we need to convert
+# our examples into Python lists. Evaluation will iterate over the lists
+print "Making the Evaluation definition"
--- End diff --
Please make all function calls Python3 compatible. The call to print.... is not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
To conclude the PR:
- I have integrated PO.DAAC data source to OCW.
- All tests run 'OK' and tests can be added further.
- I have also added an example for using podaac.
- The data source is upto date to the changes occured recently in PO.DAAC WS, which affected the data source to be integrated into OCW at a very minute level and these differences have been addressed.
@lewismc can I get a '+1' for merge now? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc Please take a look at the above change.
This [cheat sheet](http://python-future.org/compatible_idioms.html) could come in handy for in future for checking the differences between python 2 and 3. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452072
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
+import numpy as np
+from ocw.dataset import Dataset
+from netCDF4 import Dataset as netcdf_dataset
+from netcdftime import utime
+import os
+import urllib
+import xml.etree.ElementTree as ET
+
+
+def convert_times_to_datetime(time):
+ '''Convert the time object's values to datetime objects
+
+ The time values are stored as some unit since an epoch. These need to be
+ converted into datetime objects for the OCW Dataset object.
+
+ :param time: The time object's values to convert
+ :type time: pydap.model.BaseType
+
+ :returns: list of converted time values as datetime objects
+ '''
+ units = time.units
+ # parse the time units string into a useful object.
+ # NOTE: This assumes a 'standard' calendar. It's possible (likely?) that
+ # users will want to customize this in the future.
+ parsed_time = utime(units)
+ return [parsed_time.num2date(x) for x in time[:]]
+
+
+def load_dataset(variable, datasetId='', datasetShortName='', name=''):
+ '''Loads a Dataset from PODAAC
+
+ :param variable: The name of the variable to read from the dataset.
+ :type variable: :mod:`string`
+
+ :param datasetId: dataset persistent ID. datasetId or \
+ shortName is required for a granule search. Example: \
+ PODAAC-ASOP2-25X01
+ :type datasetId: :mod:`string`
+
+ :param shortName: the shorter name for a dataset. \
+ Either shortName or datasetId is required for a \
+ granule search. Example: ASCATA-L2-25km
+ :type shortName: :mod:`string`
+
+ :param name: (Optional) A name for the loaded dataset.
+ :type name: :mod:`string`
+
+ :returns: A :class:`dataset.Dataset` containing the dataset pointed to by
+ the OpenDAP URL.
+
+ :raises: ServerError
+ '''
+ # Downloading the dataset using podaac toolkit
+ podaac = Podaac()
+ path = os.path.dirname(os.path.abspath(__file__))
+ granuleName = podaac.extract_l4_granule(
+ datasetId=datasetId, shortName=datasetShortName, path=path)
+ path = path + '/' + granuleName
+ d = netcdf_dataset(path, mode='r')
+ dataset = d.variables[variable]
+
+ # By convention, but not by standard, if the dimensions exist, they will be in the order:
+ # time (t), altitude (z), latitude (y), longitude (x)
+ # but conventions aren't always followed and all dimensions aren't always present so
+ # see if we can make some educated deductions before defaulting to just pulling the first three
+ # columns.
+ temp_dimensions = map(lambda x: x.lower(), dataset.dimensions)
+ dataset_dimensions = dataset.dimensions
+ time = dataset_dimensions[temp_dimensions.index(
+ 'time') if 'time' in temp_dimensions else 0]
+ lat = dataset_dimensions[temp_dimensions.index(
+ 'lat') if 'lat' in temp_dimensions else 1]
+ lon = dataset_dimensions[temp_dimensions.index(
+ 'lon') if 'lon' in temp_dimensions else 2]
+
+ # Time is given to us in some units since an epoch. We need to convert
--- End diff --
Make this Python function documentation rather than hidden docs please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 I'll repeat what I said above
```
@Omkar20895 can you also please squash your commits into one and edit the commit message to be "CLIMATE-769 Create data source input for NASA JPL PO.DAAC"
```
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@jarifibrahim thanks for looking into this.
@Omkar20895 please make sure that the tests are updated as is documentation along with your pull requests. We will keep reviewing this until it is stable and is well tested and the documentation reflects the newly added modules.
Please address all of the above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc roger that! I will change them. I haven't worked on this branch because the podaacpy installation is broken, please look at the latest build on this branch that has failed(look at the install-ubuntu.sh log specifically) and me and @jarifibrahim have been working on it in podaacpy, we are mostly done there, you need to add you're encrypted password as mentioned [here](https://github.com/lewismc/podaacpy/issues/57) and we are good to go. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75501372
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
+import numpy as np
+from ocw.dataset import Dataset
+from netCDF4 import Dataset as netcdf_dataset
+from netcdftime import utime
+import os
+import urllib
+import xml.etree.ElementTree as ET
+
+
+def convert_times_to_datetime(time):
+ '''Convert the time object's values to datetime objects
+
+ The time values are stored as some unit since an epoch. These need to be
+ converted into datetime objects for the OCW Dataset object.
+
+ :param time: The time object's values to convert
+ :type time: pydap.model.BaseType
+
+ :returns: list of converted time values as datetime objects
+ '''
+ units = time.units
+ # parse the time units string into a useful object.
+ # NOTE: This assumes a 'standard' calendar. It's possible (likely?) that
+ # users will want to customize this in the future.
+ parsed_time = utime(units)
+ return [parsed_time.num2date(x) for x in time[:]]
+
+
+def load_dataset(variable, datasetId='', datasetShortName='', name=''):
+ '''Loads a Dataset from PODAAC
+
+ :param variable: The name of the variable to read from the dataset.
+ :type variable: :mod:`string`
+
+ :param datasetId: dataset persistent ID. datasetId or \
+ shortName is required for a granule search. Example: \
+ PODAAC-ASOP2-25X01
+ :type datasetId: :mod:`string`
+
+ :param shortName: the shorter name for a dataset. \
+ Either shortName or datasetId is required for a \
+ granule search. Example: ASCATA-L2-25km
+ :type shortName: :mod:`string`
+
+ :param name: (Optional) A name for the loaded dataset.
+ :type name: :mod:`string`
+
+ :returns: A :class:`dataset.Dataset` containing the dataset pointed to by
+ the OpenDAP URL.
+
+ :raises: ServerError
+ '''
+ # Downloading the dataset using podaac toolkit
+ podaac = Podaac()
+ path = os.path.dirname(os.path.abspath(__file__))
+ granuleName = podaac.extract_l4_granule(
+ datasetId=datasetId, shortName=datasetShortName, path=path)
+ path = path + '/' + granuleName
+ d = netcdf_dataset(path, mode='r')
+ dataset = d.variables[variable]
+
+ # By convention, but not by standard, if the dimensions exist, they will be in the order:
+ # time (t), altitude (z), latitude (y), longitude (x)
+ # but conventions aren't always followed and all dimensions aren't always present so
+ # see if we can make some educated deductions before defaulting to just pulling the first three
+ # columns.
+ temp_dimensions = map(lambda x: x.lower(), dataset.dimensions)
+ dataset_dimensions = dataset.dimensions
+ time = dataset_dimensions[temp_dimensions.index(
+ 'time') if 'time' in temp_dimensions else 0]
+ lat = dataset_dimensions[temp_dimensions.index(
+ 'lat') if 'lat' in temp_dimensions else 1]
+ lon = dataset_dimensions[temp_dimensions.index(
+ 'lon') if 'lon' in temp_dimensions else 2]
+
+ # Time is given to us in some units since an epoch. We need to convert
+ # these values to datetime objects. Note that we use the main object's
+ # time object and not the dataset specific reference to it. We need to
+ # grab the 'units' from it and it fails on the dataset specific object.
+ times = np.array(convert_times_to_datetime(d[time]))
+ lats = np.array(d.variables[lat][:])
+ lons = np.array(d.variables[lon][:])
+ values = np.array(dataset[:])
+ origin = {
+ 'source': 'PO.DAAC',
+ 'url': 'podaac.jpl.nasa.gov/ws'
+ }
+
+ # Removing the downloaded temporary granule before creating the OCW
--- End diff --
@lewismc these comments are code specific implementation information which is generally abstracted from the user, this was just for improving the readability of the code. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75535823
--- Diff: ocw/tests/test_podaac.py ---
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import ocw.data_source.podaac as podaac
+import unittest
+import os
+from ocw.dataset import Dataset
+
+
+class TestPodaacDataSource(unittest.TestCase):
--- End diff --
OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 please finish up this implementation as soon as possible. There is very little to do and the work seems to have stalled for some reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by agoodm <gi...@git.apache.org>.
Github user agoodm commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 Let me know if you need help with this. For the conda build you'll need to make a separate recipe for `podaacpy` since there is no recipe for that in the default channels. If you want I can update the recipes directory in a separate PR. If you would prefer to do it yourself in this one, you can easily make the recipe files using:
```
conda skeleton pypi podaacpy
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 can you make sure that all of this code is PEP8 formatted? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
Pheww! The PR is merged and with this my GSOC would have come to an end, but obviously PO.DAAC JPL doesn't want that(:P). Anyways it has been a really good learning experience and I would definitely continue to contribute to this project.
@lewismc Thanks for being such a good mentor.
P.S : @agoodm we need to starting working on making a conda recipe for podaacpy. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@huikyole if the tests are broken then we are not merging... that doesn't make sense.
@Omkar20895
Please log issues for both
> Outdated theme in the documentation.
and
>Broken documentation in dap.py,
Please submit pull requests in addition to this PR such that the tests become stable in TravisCI.
Once that is done, please open Tickets for both of the above issues you;ve highlighted and I will work with you to resolve them.
Thanks @Omkar20895 good work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc you are right, the search_granule has been compromised \U0001f605 , they have removed the shortName option from it, this PR is gonna take a while to get merged now. we need to update podaacpy and release a new version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452120
--- Diff: ocw/tests/test_podaac.py ---
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import ocw.data_source.podaac as podaac
+import unittest
+import os
+from ocw.dataset import Dataset
+
+
+class TestPodaacDataSource(unittest.TestCase):
--- End diff --
Where is the documentation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc I think the new version luckily does not affect the present behaviour of podaac_datasource since podaac_datasource depends solely on search_granule and dap which are intact in the new version compared to the previous version. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75575915
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
--- End diff --
@jarifibrahim It should be from podaac.podaac_data_source import Podaac. I will change it. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 can you also please squash your commits into one and edit the commit message to be "CLIMATE-769 Create data source input for NASA JPL PO.DAAC"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452231
--- Diff: ocw/tests/test_podaac.py ---
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import ocw.data_source.podaac as podaac
+import unittest
+import os
+from ocw.dataset import Dataset
+
+
+class TestPodaacDataSource(unittest.TestCase):
--- End diff --
Does the function do cleanup? Yes or no? If not then it need to clean up. Thanks @Omkar20895
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc I got that earlier, i had some merge conflicts while I was squashing and it was late at night, so I figured I would solve them in the morning, I will work on it now, sorry for the delay, Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r76349961
--- Diff: easy-ocw/ocw-pip-dependencies.txt ---
@@ -9,3 +9,4 @@ esgf-pyclient>=0.1.6
python-dateutil>=2.5.3
mock>=2.0.0
myproxyclient>=1.4.3
+podaacpy>=1.0.1
--- End diff --
Once this is upgraded to 1.0.2 we are good to merge this patch. Thanks @Omkar20895
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by agoodm <gi...@git.apache.org>.
Github user agoodm commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 @lewismc Do you know how we will handle this as a dependency? If we opt to not ship the source code from `podaac_data_source` with OCW, then we will need to add a new conda recipe. We can easily handle this if the code is on PyPi or Github.
We should also consider adding the podaac loader to the list of default loaders in `dataset_loader.py` once that's sorted out. You will need to update the comments in the class constructor as well as add a reference to the loader function in the `_source_loaders` dict.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
Thanks @agoodm good man
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
+1 please do.
BTW @Omkar20895 did you check out https://github.com/lewismc/podaacpy/issues/64
LOL
We should still commit and improve the podaac_datasource based upon what does and does not work within the PO.DAAC WS API
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r76080409
--- Diff: ocw/dataset_loader.py ---
@@ -24,6 +24,7 @@
import ocw.data_source.esgf as esgf
import ocw.data_source.rcmed as rcmed
import ocw.data_source.dap as dap
+import ocw.data_source.podaac as podaac
--- End diff --
Should be ``` +import ocw.data_source.podaac_datasource as podaac ````
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by huikyole <gi...@git.apache.org>.
Github user huikyole commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 @lewismc Would it be easier if we merge this PR anyway and updating the tests later?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75452084
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
+import numpy as np
+from ocw.dataset import Dataset
+from netCDF4 import Dataset as netcdf_dataset
+from netcdftime import utime
+import os
+import urllib
+import xml.etree.ElementTree as ET
+
+
+def convert_times_to_datetime(time):
+ '''Convert the time object's values to datetime objects
+
+ The time values are stored as some unit since an epoch. These need to be
+ converted into datetime objects for the OCW Dataset object.
+
+ :param time: The time object's values to convert
+ :type time: pydap.model.BaseType
+
+ :returns: list of converted time values as datetime objects
+ '''
+ units = time.units
+ # parse the time units string into a useful object.
+ # NOTE: This assumes a 'standard' calendar. It's possible (likely?) that
+ # users will want to customize this in the future.
+ parsed_time = utime(units)
+ return [parsed_time.num2date(x) for x in time[:]]
+
+
+def load_dataset(variable, datasetId='', datasetShortName='', name=''):
+ '''Loads a Dataset from PODAAC
+
+ :param variable: The name of the variable to read from the dataset.
+ :type variable: :mod:`string`
+
+ :param datasetId: dataset persistent ID. datasetId or \
+ shortName is required for a granule search. Example: \
+ PODAAC-ASOP2-25X01
+ :type datasetId: :mod:`string`
+
+ :param shortName: the shorter name for a dataset. \
+ Either shortName or datasetId is required for a \
+ granule search. Example: ASCATA-L2-25km
+ :type shortName: :mod:`string`
+
+ :param name: (Optional) A name for the loaded dataset.
+ :type name: :mod:`string`
+
+ :returns: A :class:`dataset.Dataset` containing the dataset pointed to by
+ the OpenDAP URL.
+
+ :raises: ServerError
+ '''
+ # Downloading the dataset using podaac toolkit
+ podaac = Podaac()
+ path = os.path.dirname(os.path.abspath(__file__))
+ granuleName = podaac.extract_l4_granule(
+ datasetId=datasetId, shortName=datasetShortName, path=path)
+ path = path + '/' + granuleName
+ d = netcdf_dataset(path, mode='r')
+ dataset = d.variables[variable]
+
+ # By convention, but not by standard, if the dimensions exist, they will be in the order:
+ # time (t), altitude (z), latitude (y), longitude (x)
+ # but conventions aren't always followed and all dimensions aren't always present so
+ # see if we can make some educated deductions before defaulting to just pulling the first three
+ # columns.
+ temp_dimensions = map(lambda x: x.lower(), dataset.dimensions)
+ dataset_dimensions = dataset.dimensions
+ time = dataset_dimensions[temp_dimensions.index(
+ 'time') if 'time' in temp_dimensions else 0]
+ lat = dataset_dimensions[temp_dimensions.index(
+ 'lat') if 'lat' in temp_dimensions else 1]
+ lon = dataset_dimensions[temp_dimensions.index(
+ 'lon') if 'lon' in temp_dimensions else 2]
+
+ # Time is given to us in some units since an epoch. We need to convert
+ # these values to datetime objects. Note that we use the main object's
+ # time object and not the dataset specific reference to it. We need to
+ # grab the 'units' from it and it fails on the dataset specific object.
+ times = np.array(convert_times_to_datetime(d[time]))
+ lats = np.array(d.variables[lat][:])
+ lons = np.array(d.variables[lon][:])
+ values = np.array(dataset[:])
+ origin = {
+ 'source': 'PO.DAAC',
+ 'url': 'podaac.jpl.nasa.gov/ws'
+ }
+
+ # Removing the downloaded temporary granule before creating the OCW
--- End diff --
Make this Python function documentation rather than hidden docs please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
That's odd, the tests failed in dap.py and I haven't even touched dap.py in this PR.
@lewismc @jarifibrahim
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r76080544
--- Diff: ocw/tests/test_podaac.py ---
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import ocw.data_source.podaac as podaac
--- End diff --
Should be ``` +import ocw.data_source.podaac_datasource as podaac ```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@agoodm yes definitely! As soon as this PR is merged let's open another PR and work on setting up a conda recipe for podaac. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@agoodm we are pushing the third party library to Pypi. It won't be an issue. This PR is just a start and is not ready yet by any means. Thanks fro reviewing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/climate/pull/383
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc I think the failure of the tests is due to some other module but not podaac, I have run the tests cases in my local machine all the tests related to podaac data source are running OK. Thanks.
@huikyole I agree with you, the branch is quite stable with just one documentation update on its way. With that update I think we can merge this branch. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
The documentation of dap.py is broken and also when I try to build the documentation locally I get the error, theme not found. Someone please check this. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/climate/pull/383
[](https://coveralls.io/builds/7631290)
Coverage remained the same at 80.617% when pulling **39960b5d82c86126a10d7c0c119d98418294f654 on Omkar20895:CLIMATE-769** into **91ec7f31c0c1cd6ac77372de7409f070f5d1e723 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
In this latest commit I have added,
- An example on usage of podaac data source which evaluates Temporal STD metric, the example on running generates a contour map and I have cross checked it to the map generated in panoply.
- A documentation update to podaac data source.
Issues to be addressed
- Outdated theme in the documentation.
- Broken documentation in dap.py, the
> _convert_times_to_datetime() function is dropped by automodule of sphinx and hence does not appear in the documentation.
@lewismc @huikyole @jarifibrahim
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
Finally!! The build passed OK! \U0001f604 @lewismc I will add in a commit and merge it, Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@agoodm please reach out to us for any issues.
@Omkar20895 excellent work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75455700
--- Diff: ocw/tests/test_podaac.py ---
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import ocw.data_source.podaac as podaac
+import unittest
+import os
+from ocw.dataset import Dataset
+
+
+class TestPodaacDataSource(unittest.TestCase):
--- End diff --
@lewismc Yes the function does the clean up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
Cool, if travis-ci build is stable then please merge with master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r76080210
--- Diff: examples/podaac_integration_example.py ---
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import ocw.data_source.podaac as podaac
--- End diff --
Should be ``` +import ocw.data_source.podaac_datasource as podaac
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 please add Podaacpy to the relevant build configuration
Current version is 1.0.1
https://testpypi.python.org/pypi/podaacpy
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by OCWJenkins <gi...@git.apache.org>.
Github user OCWJenkins commented on the issue:
https://github.com/apache/climate/pull/383
Merged build triggered. Test Failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by jarifibrahim <gi...@git.apache.org>.
Github user jarifibrahim commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc The exception can be fixed by renaming `time_range` to `temporal_boundaries`.
But there are many more exceptions in the unit tests. A lot of new code has been added lately but the tests have not been updated. The tests were failing and still the code was merged. I'm working on fixing these tests. Please do not merge until the tests are fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@lewismc I will push a trivial update as soon as the podaacpy installation becomes stable and validate the conda installation after integrating this. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/climate/pull/383
@Omkar20895 here you go
https://pypi.python.org/pypi/podaacpy
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/climate/pull/383
[](https://coveralls.io/builds/7616423)
Coverage remained the same at 80.622% when pulling **0a9d6b4ff80d0d03aeec019030e5bcdd187e8f55 on Omkar20895:CLIMATE-769** into **91ec7f31c0c1cd6ac77372de7409f070f5d1e723 on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate issue #383: Adding PO.DAAC data source along with the tests
Posted by Omkar20895 <gi...@git.apache.org>.
Github user Omkar20895 commented on the issue:
https://github.com/apache/climate/pull/383
@agoodm thank you very much!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] climate pull request #383: Adding PO.DAAC data source along with the tests
Posted by jarifibrahim <gi...@git.apache.org>.
Github user jarifibrahim commented on a diff in the pull request:
https://github.com/apache/climate/pull/383#discussion_r75575893
--- Diff: ocw/data_source/podaac.py ---
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from podaac_data_source import Podaac
--- End diff --
@Omkar20895 This needs to be `from podaacpy.podaac_data_source import Podaac `.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---