You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@climate.apache.org by agoodm <gi...@git.apache.org> on 2016/07/25 23:35:25 UTC

[GitHub] climate pull request #374: CLIMATE-825 - Coalesce data sources into one modu...

GitHub user agoodm opened a pull request:

    https://github.com/apache/climate/pull/374

    CLIMATE-825 - Coalesce data sources into one module

    -Added a new class called `DatasetLoader`
    -Added corresponding documentation and unit tests
    
    Read the docstrings for more info on the API, but the gist of it is that it was designed to easily load the inputs to each data source loader from our current config files.
    
    This is intended for the 1.2 release.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/agoodm/climate CLIMATE-825

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/climate/pull/374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #374
    
----
commit ecea6210b717e9464ce527c883286a38e41a7a08
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-18T17:11:13Z

    Added DatasetLoader class

commit a3764c046adb506740fc7e3abdbabedfea917ad5
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-18T22:11:41Z

    Added additional functionality to DatasetLoader

commit bf5b6eba6251c54317693cc1df2a05e3e8401dfb
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-19T20:38:14Z

    Minor fixes to dataset_loader

commit d937675d0ad1da865d5dab57d895cf77e471f646
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-19T20:42:25Z

    Minor docstring fix

commit 4add58b2e5dd4e6b28509402e35adffbc03692e7
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-25T22:14:42Z

    Preserve data_source info after each load

commit 198de48332c84586dafcaf033fa27a8eb8818ebe
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-25T23:13:48Z

    Fixing some bugs found from testing

commit 98a67d130ad9b2c946f18dba90da6b68c51594de
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-25T23:14:14Z

    Added unit tests

commit b56b690cb21b260d146a6080fff33b9c0c77319e
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-25T23:21:38Z

    Add ASF Licence header to test_dataset_loader.py

commit 541dbe53fcb4d014161d9a64f7b8f890f3360069
Author: Alex Goodman <ag...@users.noreply.github.com>
Date:   2016-07-25T23:29:28Z

    Added dataset_loader.py to Sphinx build

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] climate issue #374: CLIMATE-825 - Coalesce data sources into one module

Posted by coveralls <gi...@git.apache.org>.

Github user coveralls commented on the issue:

    https://github.com/apache/climate/pull/374
  
    
    [![Coverage Status](https://coveralls.io/builds/7159704/badge)](https://coveralls.io/builds/7159704)
    
    Coverage increased (+0.6%) to 84.256% when pulling **541dbe53fcb4d014161d9a64f7b8f890f3360069 on agoodm:CLIMATE-825** into **442a9b2d7ef0c42547f1a055a513b07face41553 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] climate issue #374: CLIMATE-825 - Coalesce data sources into one module

Posted by agoodm <gi...@git.apache.org>.

Github user agoodm commented on the issue:

https://github.com/apache/climate/pull/374

@huikyole Not sure if I agree with your first suggestion. I thought the intent of having data_source as a separate directory was originally due to having multiple modules for loading from each data source. The modules in the base ocw directory represent each of the individual steps in the workflow, eg dataset processing, running evaluations, and plotting. The main thing that was missing previously was loading the datasets which is exactly what this module aims to do. So for now I think leaving it here is appropriate. I think @lewismc should share his thoughts on this though too.

I absolutely agree with your second suggestion though. I originally had it set up this way because to my recollection, the rest of the OCW codebase (particularly metrics and evaluations) were designed with "one reference" dataset. Given that Loikith et al. 2013 uses two reanalysis datasets, we should get rid of this rigid assumption not only for `dataset_loader.py` but potentially for `evaluation.py` as well. I think for now changing the former obviously takes precedence but we should consider exploring the the latter as well.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] climate issue #374: CLIMATE-825 - Coalesce data sources into one module

Posted by jarifibrahim <gi...@git.apache.org>.

Github user jarifibrahim commented on the issue:

    https://github.com/apache/climate/pull/374
  
    @agoodm I'm not sure why the build is stuck. I tried starting a new build but it also got stuck at `waiting`. The issue seems to be with landscape. Please go ahead and merge this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] climate issue #374: CLIMATE-825 - Coalesce data sources into one module

Posted by huikyole <gi...@git.apache.org>.

Github user huikyole commented on the issue:

    https://github.com/apache/climate/pull/374
  
    @agoodm This is wonderful. Thank you so much. I have very minor suggestions.
    - dataset_loader.py can be located in ocw/data_source folder.
    - Would it be possible to have 1) multiple references and 2) reference only without targets?
    1) is very useful when calculating metrics using multiple variables. 
    2) for example, we should be able to load daily reanalysis data only and calculate k-means clustering.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] climate issue #374: CLIMATE-825 - Coalesce data sources into one module

Posted by jarifibrahim <gi...@git.apache.org>.

Github user jarifibrahim commented on the issue:

    https://github.com/apache/climate/pull/374
  
    @agoodm can you please fix landscape.io issues? The build failed because there were multiple PEP8 violations. Please take a look at this -> [Link](https://landscape.io/diff/413629)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---