You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> on 2013/01/26 20:48:05 UTC

[DISCUSS] Apache Open Climate Workbench

Hi Everyone!

I bring to the Incubator a new proposal for the Apache Open Climate
Workbench (incubating) project. I've added the proposal to the Incubator
wiki here:

http://wiki.apache.org/incubator/ClimateProposal


The project is a distributed, scalable approach for the rapid comparison
of remote sensing data (e.g., from NASA) with that of climate model output
generated by major US and international activities including the US
National Climate Assessment, the Coordinated Regional Downscaling
Experiment (CORDEX), the North American Regional Climate Change Assessment
Program (NARCCAP) the Intergovernmental Panel on Climate Change (IPCC).
Much of the software to be donated and evolved here at Apache as a
community includes core dependencies to several Apache projects (OODT,
Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.

We welcome your feedback over the next week or so for discussion and look
forward to it!

Cheers,
Chris Mattmann
Proposed Mentor and Champion

---------------proposal text

= Apache Open Climate Workbench, tool for scalable comparison of remote
sensing observations to climate model outputs, regionally and globally. =
=== Abstract ===
The Apache Open Climate Workbench proposal desires to contribute an
existing community of software related to the analysis and evaluation of
climate models, and related to the use of remote sensing data in that
process. 

Specifically, we will bring a fundamental software toolkit for analysis
and evaluation of climate model output against remote sensing data. The
toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
Evaluation System (RCMES)]]. RCMES provides two fundamental components for
the easy, intuitive comparison of climate model output against remote
sensing data. The first component called RCMED (for "Regional Climate
Model Evaluation Database") is a scalable cloud database that decimates
remote sensing data and renalysis data related to climate using Apache
OODT extractors, Apache Tika, etc. These transformations make
traditionally heterogeneous upstream remote sensing data and climate model
output homogeneous and unify them into a data point model of the form
(lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
and Longitude (lng) are in WGS84 format, but can be reformatted on the
fly. time is in ISO 8601 format, a string sortable format independent of
underlying store. value carries with it units, related to interpretation
and height allows for different values for different atmospheric vertical
levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
and Apache Hive, along with hooks to PostGIS and MySQL (traditional
relational databases). The second component of the system, RCMET (for
"Regional Climate Model Evaluation Toolkit") provides facilities for
connecting to RCMED, dynamically obtaining remote sensing data for a
space/time region of interest, grabbing associated model output (that the
user brings, or from the Earth System Grid Federation) of the same form,
and then regridding the remote sensing data to be on the model output
grid, or the model output to be on the remote sensing data grid. The
regridded data spatially is then temporally regridded using techniques
including seasonal cycle compositing (e.g., all summer months, all
Januaries, etc.), or by daily, monthly, etc. The uniform model output and
remote sensing data are then analyzed using pluggable metrics, e.g.,
Probability Distribution Functions (PDFs), Root Mean Squared Error (RMSE),
Bias, and other (possibly user-defined) techniques, computing an analyzed
comparison or evaluation. This evaluation is then visualized by plugging
in to the NCAR NCL library for producing static plots (histograms, time
series, etc.)

We also have performed a great deal of work in packaging RCMES to make the
system easy to deploy. We have working Virtual Machines (VMWare VMX and
Virtual Box OVA compatible formats) and we also have an installer built on
Python Buildout (http://buildout.org/) called "Easy RCMET" for dynamically
constructing the RCMET toolkit.

RCMES is currently supporting a number of recognized climate projects of
(inter-)national significance. In particular, RCMES is supporting the
[[http://www.globalchange.gov/what-we-do/assessment|U.S. National Climate
Assessment (NCA) activities]] on behalf of NASA's contribution to the NCA;
is working with the [[http://www.narccap.ucar.edu/|North American Regional
Climate Change Assessment Program (NARCCAP)]]; and is also working with
the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
Regional Downscaling Experiment (CORDEX)]].

=== Proposal ===
We propose to transition the RCMES software community, which includes
developers of the RCMET and RCMED software, along with users of RCMES in
the CORDEX project across a variety of academic institutions, scientists
helping to improve the RCMES metrics, and visualizations, and regridding
algorithms, packagers making RCMES easier to install, and scientists
helping to lead some of these international projects that are already
using RCMES. 

We have been working on the RCMES project since 2009 funded initially by
the American Recovery and Reinvestment Act (ARRA) project out at NASA, and
then branching out into other sources of support and sustainability (NASA;
NSF, etc. -- see the
[[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
the RCMES website for a full list of supporting U.S. and international
partners). 

With the existing RCMES community at Apache, we will also work to
encourage other climate software projects e.g., Open Climate GIS, elements
of the Earth System Grid Federation, other NASA climate projects funded
under the Computational Modeling, Algorithms and Cyberinfrastructure
(CMAC) to contribute to the Open Climate Workbench here at Apache.

RCMED is a Big Data project that combines several underlying Apache
software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related data
management software. Its primary language is Java; RCMET, on the other
hand, is a Python API, associated set of classes (framework), set of
Python Bottle Web services, and a PHP "Wizard"-based User Interface that
leverages Apache OODT Balance.

=== Background ===
Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
his solid experience with Apache OODT and bringing it to the ASF. Chris
worked for a year to get the support of the JPL community including
approvals from the Software Release authority at JPL to release the
software. 

The initial code drop will include the RCMES SVN repository from JPL
including prior revisions. We anticipate also including a smaller package,
CDX, which contains some useful facilities for regridding, and command
line tools for manipulating large datasets, and working with OPeNDAP, etc.

After the code drop, we will work with our developers, users, documentors,
and other members of the team to teach those unfamiliar with the Apache
way how it works around here at Apache. 30% of the community from RCMES
includes those intimately familiar with Apache including 5 ASF members --
the other 70% include a range of scientific code developers, climate
scientists that use RCMES, program officers that will help make
documentation and slides for the code, and advocate for it in the
community. Their experience with Apache ranges from using various ASF
products, to contributing patches to them, to not using any ASF software
at all.

With this diversity, we anticipate that while everything may not just work
turnkey out of the box, this represents a unique opportunity to
demonstrate Apache to the international community and to show the benefits
of its community and social models. That said, we also have a lot of ASF
experience to make sure everyone learns the Apache way.

=== Rationale ===
We are bringing RCMES to Apache for a few reasons. First, we feel that it
will immediately enable our collaborators across a number of institutions
both nationally and internationally have the opportunity to work on a
common software base, and to improve it with contributions from their own
sites. Currently these are difficult to negotiate now because of varied
legal and contribution frameworks -- Apache allows us to simplify this to
a unified one. Second, using the ASF's world-wide mirroring system, we
will be able to deliver climate software broadly to the community as we
release it, rather than sneaker netting the software around or
establishing our own point release infrastructure.

Bringing this project to Apache also immediately thrusts the ASF into the
thriving ecosystem of the Coordinated Regional Downscaling Experiment
(CORDEX), the US National Cimate Assessment, the North American Regional
Climate Change Assessment Program (the US contribution to CORDEX) and into
relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
assessment activities at a number of different institutions. We also seek
to help lead and encourage du jour standard development rather than top
down level dictating of standards for climate software and the ASF will
provide us a means for that.

=== Initial Goals ===
The initial goals of the proposed project are:

 * Stand up a sustaining Apache-based community around the JPL RCMES
codebase.
 * Active relationships and possible cooperation with related projects and
communities, including end user and scientific communities, CORDEX,
NARCCAP, US NCA, IPCC, ESG, etc.
 * Active relationships and possible cooperation with existing Apache
communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
 * Initial Apache release.
 * Leverage Apache Open Climate Workbench in climate activities at NASA,
in the international community as mentioned above, and beyond.
 * Vetting all software licenses and making sure IP is clear (software
grant from JPL forthcoming).
 
== Current Status ==
=== Meritocracy ===
30% of the proposed initial committers are familiar with the meritocracy
principles of Apache. As stated above this includes 5 ASF members. Of the
mentorship list, we have included Chris Douglas, a PMC member from Hadoop
and ASF member to help guide the community. Chris M. and Chris D. have
guided a number of projects through the Incubator over the years. The
other mentor includes Paul Ramirez, who has experience with the Incubator
-- he was a mentor for Apache Any23, and also was  one of the PPMC members
and eventual mentor for Apache SIS. The 70% of proposed initial committers
that aren't as familiar with Apache have a broad range of experience in
other open source projects, and have a deep respect and affinity for the
foundation and the work that gets done here. The more experience ASF
mentors and project members will help to guide them.
=== Community ===
There is an existing, established community of developers and users of
this projet. This includes established communities including the
Coordinated Regional Downscaling Experiment, the U.S. National Climate
Assessment (NCA), the North American Regional Climate Change Assessment
Program (NARCCAP), and more. The Coordinated Regional Climate Downscaling
Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
world wide effort of coordination of regional climate downscaling (RCD)
experiments driven by the World Climate Research Program (WRCP,
http://www.wcrp-climate.org/index.shtml). Recently, a large number of RCD
projects have been carried out on a large parts of the world. To maximize
the benefits of these research activities the WCRP designed a framework
(Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets of
RCD-based information for the recent historical past and 21st century
projections, covering the majority of populated land regions on the
globe".  CORDEX defined different control domains (up to 10,
http://cordex.dmi.dk/joomla/) for almost all the populated regions of the
world in a way to standardize the experiments and make them comparable. A
key region focused on Africa was also designated as the top priority by
WRCP. CORDEX also provides a a series of conventions and list of variables
that have to be followed by any project that wants to contribute to the
experiment. Each CORDEX region has a coordinator and regional and
international periodic meetings are scheduled in a way to ensure the
global well being. NARCCAP is the U.S. contribution to CORDEX. From the
[[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
National Climate Assessment]] site, work is "being conducted under the
auspices of the Global Change Research Act of 1990. The GCRA requires a
report to the President and the Congress every four years that integrates,
evaluates, and interprets the findings of the U.S. Global Change Research
Program (USGCRP); analyzes the effects of global change on the natural
environment, agriculture, energy production and use, land and water
resources, transportation, human health and welfare, human social systems,
and biological diversity; and analyzes current trends in global change,
both human-induced and natural, and projects major trends for the
subsequent 25 to 100 years."

Apache Open Climate Workbench will support all of these communities above,
with an eye towards being a general purpose climate evaluation toolkit for
model output and remote sensing data.

=== Core Developers ===
The initial set of developers comes from various NASA centers (JPL, and
Goddard Space Flight Center), NASA HQ, various  Universities participating
in CORDEX (Cape Town, University of New South Wales), the Indian Institute
of Tropical Meteorology, the Free Univ. Berlin), the University of
California Los Angeles, and Howard University. As mentioned previously
several of our developers are Apache veterans and understand how it works
around here and for those that don't, they will have great mentorship.

=== Alignment ===
Our proposed effort aligns with the U.S. National Climate Assessment, the
CORDEX effort, other efforts, including the Earth System Grid Federation,
other climate software including the Open Climate GIS toolkit, other
science portals for climate including the Climate Information Portal (CIP)
at the University of Cape Town, and other related projects.

There are also a number of related Apache projects and dependencies, that
will be mentioned in the Relationships with Other Apache products section.

== Known Risks ==
=== Orphaned products ===
Our project has a history of funding support from JPL, NASA (Applications
program/ARRA, NCA, AIST), NSF (ExArch project), international investment
from collaborators, and from other funding sources. The funding sources
are all target future deliverables and activities, so there is little
chance this software and community will be orphaned.

=== Inexperience with Open Source ===
All the initial developers have worked on open source before -- 30% of the
proposed initial community are experience with the ASF, and are PMC
members and committers on ASF project including 5 ASF members. Our mentors
are all ASF members, and we welcome any interest from additional Apache
mentors in the effort. Those 70% of our project that aren't Apache
committers, PMC members, or members will benefit from the leadership of
the other 30% of the project.

=== Homogenous Developers ===
The initial developers come from a variety of backgrounds and with a
variety of needs for the proposed framework. Everyone is used to
communicating on mailing lists as the project spans timezones,
international institutions and centers of excellence for climate science.

=== Reliance on Salaried Developers ===
All of the proposed initial developers are paid to work on this or related
projects, but the proposed project is not the primary task for anyone.

=== Relationships with Other Apache Products ===
As mentioned above, RCMES and the Apache Open Climate Workbench already
depend on Apache OODT for facade interfaces to underlying data warehouses
for storing remote sensing data; and for metadata extraction and
transformation. The software also uses Apache Tika for this (through a
transitive dependency from OODT). In addition, we have hooks to Apache
Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
remote sensing data from MySQL and into HIVE.

=== A Excessive Fascination with the Apache Brand ===
All of us are familiar with Apache and have a respect for its brand and
community. Chris Mattmann is a big proponent of Apache's sustainability
factor -- and it's ability to grow software communities, in an
institution, or funding source neutral manner. All of the community have
an extreme respect for Apache, including those in our communities who
aren't necessarily trained computer scientists, but are Scientists (big
"S", e.g., land, physical, Earth/Climate scientists).

== Documentation ==
The initial RCMES code base will come from the internal JPL Subversion
repository. The [[Regional Climate Model Evaluation System (RCMES)
project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
has documentation on the existing software, including links to funding
support, communities, and other projects. We will continue to maintain
that site at JPL, which is part of the reason for rebranding the project
here at Apache with a new name to not interfere with the existing RCMES
one that has a following. In addition, we hope to evolve RCMES@JPL to have
increasing levels of dependency on Apache Open Climate Workbench, so that
we can incrementally transition with little impact to existing customers.

In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange (CDX)]]
website also has documentation on the existing software.

== Initial Source ==
The project will start with seed code donated by NASA JPL via Mattmann and
the rest of the initial committers, which consists of the Regional Climate
Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
(CDX) software. This will include the core Python API for RCMET, the RCMED
OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
and HIVE), and the RCMED extractors for various climate formats. The
source will also include Easy-RCMET, the Python Buildout for RCMET. In
addition, we will bring along the CDX toolkit, which includes a CDX client
package that performs subsetting, access, regridding of climate data; and
also includes a Python Buildout installer of its own called Uber CDX.

== Source and Intellectual Property Submission Plan ==
All seed code and other contributions will be handled through the normal
Apache contribution process. Mattmann has been authorized by NASA JPL to
lead the contribution of RCMES and CDX into the Incubator via his existing
Apache CLA, and a Software Grant to be provided.

We will also contact other related efforts for possible cooperation and
contributions.

== External Dependencies ==
Our project depends on a number of external libraries with various
licensing conditions. An initial list of such dependencies is shown below.
||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT compat||
||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modified
 PSF license||
||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||MI
T||


== Cryptography ==
The project itself will not use cryptography, but it is possible that some
of the external software libraries will include cryptographic code to
handle features present in various science data formats. If we need to
provide an export control statement regarding cryptographic code per
Apache policy, we will follow a similar approach by Mattmann in
[[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting lead
this effort in Apache Tika. Mattmann is familiar with this process.

== Required Resources ==
Mailing lists

 * dev@climate.incubator.apache.org
 * commits@climate.incubator.apache.org
 * private@climate.incubator.apache.org

Subversion Directory

 * https://svn.apache.org/repos/asf/incubator/climate

Issue Tracking

 * JIRA CLIMATE (CLIMATE)

Other Resources

 * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
 * Review Board instance - CLIMATE
 * Jenkins instance - CLIMATE

== Initial Committers ==
||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
||Chris A. Mattmann ||mattmann at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Cameron E. Goodale ||cgoodale at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Paul Ramirez ||pramirez at apache dog org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Andrew F. Hart ||ahart at apache dot org
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
||Jinwon Kim||jkim at atmos dot ucla dot edu
||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
System Science and Engineering]] || no||
||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
||Daniel J. Crichton||crichton at apache dot
org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.html
|Howard University]] || no ||
||Paul Zimdars||pzimdars at apache dot
org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||Chris Jack||cjack at csag dot uct dot ac dot
za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
||Bruce Hewitson||hewitson at csag dot uct dot ac dot
za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
||Jason Evans||jason dot evans at unsw dot edu dot
au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
||J. Sanjay||sanjay at tropmet dot res dot in ||
[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || yes
||
||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
yes ||
||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
[[http://hq.nassa.gov/|NASA HQ]] || no ||
||Laura Carriere||laura dot carriere at nasa dot gov
||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no ||
||Denis Nadeau|| denis dot nadeau at nasa dot
gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
||


== Sponsors ==
Champion

 * Chris Mattmann (mattmann at apache dot org)

Nominated Mentors

 * Chris A. Mattmann (mattmann at apache dot org)
 * Chris Douglas (cdouglas at apache dot org)
 * Paul Ramirez (pramirez at apache dot org)

Sponsoring Entity

 * Apache Incubator



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Suresh,

We would be honored to have you on board. Thank you and I'm looking
forward to your help and guidance!

Cheers,
Chris

P.S. Add yourself to the wiki, no problem!

On 1/27/13 4:36 PM, "Suresh Marru" <sm...@apache.org> wrote:

>
>On Jan 27, 2013, at 1:56 PM, "Mattmann, Chris A (388J)"
><ch...@jpl.nasa.gov> wrote:
>
>> Hey Suresh,
>> 
>> Thanks a ton man and great to hear!
>> 
>> To your comment below:
>> 
>> On 1/27/13 5:56 AM, "Suresh Marru" <sm...@apache.org> wrote:
>> 
>>> Hi Chris,
>>> 
>>> Great proposal, I am personally looking forward to see more of these
>>> federal government and international collaboration efforts adopt open
>>> community process. Impressive to see this project is pulling together
>>> initial developers from government and universities of US, South
>>>Africa,
>>> UK, Germany and India (I will hope all of them will be able to get
>>>their
>>> CLA's cleared). You are really scraping the black ice on bureaucratic
>>> highways and preventing at least some reinventing wheels with valuable
>>> tax payers money in multiple countries.
>>> 
>>> Will this system evaluate and be inclusive of the community earth
>>>system
>>> modeling community?
>> 
>> Yep this proposal includes some of those that are directly involved in
>> that community including Tsengdar Lee, as well as members of the Earth
>> System Grid Federation. We will definitely be inclusive and are happy to
>> have involvement.
>
>Sounds very good. I am unable to resist my itch, so will like to jump on
>the band wagon as a committer. Please let me know if I can add myself to
>the proposal wiki.
>
>Thanks,
>Suresh
>
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>>> 
>>> Suresh
>>> 
>>> On Jan 26, 2013, at 2:48 PM, "Mattmann, Chris A (388J)"
>>> <ch...@jpl.nasa.gov> wrote:
>>> 
>>>> Hi Everyone!
>>>> 
>>>> I bring to the Incubator a new proposal for the Apache Open Climate
>>>> Workbench (incubating) project. I've added the proposal to the
>>>>Incubator
>>>> wiki here:
>>>> 
>>>> http://wiki.apache.org/incubator/ClimateProposal
>>>> 
>>>> 
>>>> The project is a distributed, scalable approach for the rapid
>>>>comparison
>>>> of remote sensing data (e.g., from NASA) with that of climate model
>>>> output
>>>> generated by major US and international activities including the US
>>>> National Climate Assessment, the Coordinated Regional Downscaling
>>>> Experiment (CORDEX), the North American Regional Climate Change
>>>> Assessment
>>>> Program (NARCCAP) the Intergovernmental Panel on Climate Change
>>>>(IPCC).
>>>> Much of the software to be donated and evolved here at Apache as a
>>>> community includes core dependencies to several Apache projects (OODT,
>>>> Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
>>>> software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.
>>>> 
>>>> We welcome your feedback over the next week or so for discussion and
>>>> look
>>>> forward to it!
>>>> 
>>>> Cheers,
>>>> Chris Mattmann
>>>> Proposed Mentor and Champion
>>>> 
>>>> ---------------proposal text
>>>> 
>>>> = Apache Open Climate Workbench, tool for scalable comparison of
>>>>remote
>>>> sensing observations to climate model outputs, regionally and
>>>>globally.
>>>> =
>>>> === Abstract ===
>>>> The Apache Open Climate Workbench proposal desires to contribute an
>>>> existing community of software related to the analysis and evaluation
>>>>of
>>>> climate models, and related to the use of remote sensing data in that
>>>> process. 
>>>> 
>>>> Specifically, we will bring a fundamental software toolkit for
>>>>analysis
>>>> and evaluation of climate model output against remote sensing data.
>>>>The
>>>> toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate
>>>>Model
>>>> Evaluation System (RCMES)]]. RCMES provides two fundamental components
>>>> for
>>>> the easy, intuitive comparison of climate model output against remote
>>>> sensing data. The first component called RCMED (for "Regional Climate
>>>> Model Evaluation Database") is a scalable cloud database that
>>>>decimates
>>>> remote sensing data and renalysis data related to climate using Apache
>>>> OODT extractors, Apache Tika, etc. These transformations make
>>>> traditionally heterogeneous upstream remote sensing data and climate
>>>> model
>>>> output homogeneous and unify them into a data point model of the form
>>>> (lat, lng, time, value, height) on a per parameter basis. Latitude
>>>>(lat)
>>>> and Longitude (lng) are in WGS84 format, but can be reformatted on the
>>>> fly. time is in ISO 8601 format, a string sortable format independent
>>>>of
>>>> underlying store. value carries with it units, related to
>>>>interpretation
>>>> and height allows for different values for different atmospheric
>>>> vertical
>>>> levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache
>>>>Hadoop
>>>> and Apache Hive, along with hooks to PostGIS and MySQL (traditional
>>>> relational databases). The second component of the system, RCMET (for
>>>> "Regional Climate Model Evaluation Toolkit") provides facilities for
>>>> connecting to RCMED, dynamically obtaining remote sensing data for a
>>>> space/time region of interest, grabbing associated model output (that
>>>> the
>>>> user brings, or from the Earth System Grid Federation) of the same
>>>>form,
>>>> and then regridding the remote sensing data to be on the model output
>>>> grid, or the model output to be on the remote sensing data grid. The
>>>> regridded data spatially is then temporally regridded using techniques
>>>> including seasonal cycle compositing (e.g., all summer months, all
>>>> Januaries, etc.), or by daily, monthly, etc. The uniform model output
>>>> and
>>>> remote sensing data are then analyzed using pluggable metrics, e.g.,
>>>> Probability Distribution Functions (PDFs), Root Mean Squared Error
>>>> (RMSE),
>>>> Bias, and other (possibly user-defined) techniques, computing an
>>>> analyzed
>>>> comparison or evaluation. This evaluation is then visualized by
>>>>plugging
>>>> in to the NCAR NCL library for producing static plots (histograms,
>>>>time
>>>> series, etc.)
>>>> 
>>>> We also have performed a great deal of work in packaging RCMES to make
>>>> the
>>>> system easy to deploy. We have working Virtual Machines (VMWare VMX
>>>>and
>>>> Virtual Box OVA compatible formats) and we also have an installer
>>>>built
>>>> on
>>>> Python Buildout (http://buildout.org/) called "Easy RCMET" for
>>>> dynamically
>>>> constructing the RCMET toolkit.
>>>> 
>>>> RCMES is currently supporting a number of recognized climate projects
>>>>of
>>>> (inter-)national significance. In particular, RCMES is supporting the
>>>> [[http://www.globalchange.gov/what-we-do/assessment|U.S. National
>>>> Climate
>>>> Assessment (NCA) activities]] on behalf of NASA's contribution to the
>>>> NCA;
>>>> is working with the [[http://www.narccap.ucar.edu/|North American
>>>> Regional
>>>> Climate Change Assessment Program (NARCCAP)]]; and is also working
>>>>with
>>>> the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
>>>> Regional Downscaling Experiment (CORDEX)]].
>>>> 
>>>> === Proposal ===
>>>> We propose to transition the RCMES software community, which includes
>>>> developers of the RCMET and RCMED software, along with users of RCMES
>>>>in
>>>> the CORDEX project across a variety of academic institutions,
>>>>scientists
>>>> helping to improve the RCMES metrics, and visualizations, and
>>>>regridding
>>>> algorithms, packagers making RCMES easier to install, and scientists
>>>> helping to lead some of these international projects that are already
>>>> using RCMES. 
>>>> 
>>>> We have been working on the RCMES project since 2009 funded initially
>>>>by
>>>> the American Recovery and Reinvestment Act (ARRA) project out at NASA,
>>>> and
>>>> then branching out into other sources of support and sustainability
>>>> (NASA;
>>>> NSF, etc. -- see the
>>>> [[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section
>>>>on
>>>> the RCMES website for a full list of supporting U.S. and international
>>>> partners). 
>>>> 
>>>> With the existing RCMES community at Apache, we will also work to
>>>> encourage other climate software projects e.g., Open Climate GIS,
>>>> elements
>>>> of the Earth System Grid Federation, other NASA climate projects
>>>>funded
>>>> under the Computational Modeling, Algorithms and Cyberinfrastructure
>>>> (CMAC) to contribute to the Open Climate Workbench here at Apache.
>>>> 
>>>> RCMED is a Big Data project that combines several underlying Apache
>>>> software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related
>>>> data
>>>> management software. Its primary language is Java; RCMET, on the other
>>>> hand, is a Python API, associated set of classes (framework), set of
>>>> Python Bottle Web services, and a PHP "Wizard"-based User Interface
>>>>that
>>>> leverages Apache OODT Balance.
>>>> 
>>>> === Background ===
>>>> Bringing RCMES to Apache was the brain-child of Chris Mattmann, based
>>>>on
>>>> his solid experience with Apache OODT and bringing it to the ASF.
>>>>Chris
>>>> worked for a year to get the support of the JPL community including
>>>> approvals from the Software Release authority at JPL to release the
>>>> software. 
>>>> 
>>>> The initial code drop will include the RCMES SVN repository from JPL
>>>> including prior revisions. We anticipate also including a smaller
>>>> package,
>>>> CDX, which contains some useful facilities for regridding, and command
>>>> line tools for manipulating large datasets, and working with OPeNDAP,
>>>> etc.
>>>> 
>>>> After the code drop, we will work with our developers, users,
>>>> documentors,
>>>> and other members of the team to teach those unfamiliar with the
>>>>Apache
>>>> way how it works around here at Apache. 30% of the community from
>>>>RCMES
>>>> includes those intimately familiar with Apache including 5 ASF members
>>>> --
>>>> the other 70% include a range of scientific code developers, climate
>>>> scientists that use RCMES, program officers that will help make
>>>> documentation and slides for the code, and advocate for it in the
>>>> community. Their experience with Apache ranges from using various ASF
>>>> products, to contributing patches to them, to not using any ASF
>>>>software
>>>> at all.
>>>> 
>>>> With this diversity, we anticipate that while everything may not just
>>>> work
>>>> turnkey out of the box, this represents a unique opportunity to
>>>> demonstrate Apache to the international community and to show the
>>>> benefits
>>>> of its community and social models. That said, we also have a lot of
>>>>ASF
>>>> experience to make sure everyone learns the Apache way.
>>>> 
>>>> === Rationale ===
>>>> We are bringing RCMES to Apache for a few reasons. First, we feel that
>>>> it
>>>> will immediately enable our collaborators across a number of
>>>> institutions
>>>> both nationally and internationally have the opportunity to work on a
>>>> common software base, and to improve it with contributions from their
>>>> own
>>>> sites. Currently these are difficult to negotiate now because of
>>>>varied
>>>> legal and contribution frameworks -- Apache allows us to simplify this
>>>> to
>>>> a unified one. Second, using the ASF's world-wide mirroring system, we
>>>> will be able to deliver climate software broadly to the community as
>>>>we
>>>> release it, rather than sneaker netting the software around or
>>>> establishing our own point release infrastructure.
>>>> 
>>>> Bringing this project to Apache also immediately thrusts the ASF into
>>>> the
>>>> thriving ecosystem of the Coordinated Regional Downscaling Experiment
>>>> (CORDEX), the US National Cimate Assessment, the North American
>>>>Regional
>>>> Climate Change Assessment Program (the US contribution to CORDEX) and
>>>> into
>>>> relevance for upcoming Intergovernmental Panel on Climate Change
>>>>(IPCC)
>>>> assessment activities at a number of different institutions. We also
>>>> seek
>>>> to help lead and encourage du jour standard development rather than
>>>>top
>>>> down level dictating of standards for climate software and the ASF
>>>>will
>>>> provide us a means for that.
>>>> 
>>>> === Initial Goals ===
>>>> The initial goals of the proposed project are:
>>>> 
>>>> * Stand up a sustaining Apache-based community around the JPL RCMES
>>>> codebase.
>>>> * Active relationships and possible cooperation with related projects
>>>> and
>>>> communities, including end user and scientific communities, CORDEX,
>>>> NARCCAP, US NCA, IPCC, ESG, etc.
>>>> * Active relationships and possible cooperation with existing Apache
>>>> communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
>>>> * Initial Apache release.
>>>> * Leverage Apache Open Climate Workbench in climate activities at
>>>>NASA,
>>>> in the international community as mentioned above, and beyond.
>>>> * Vetting all software licenses and making sure IP is clear (software
>>>> grant from JPL forthcoming).
>>>> 
>>>> == Current Status ==
>>>> === Meritocracy ===
>>>> 30% of the proposed initial committers are familiar with the
>>>>meritocracy
>>>> principles of Apache. As stated above this includes 5 ASF members. Of
>>>> the
>>>> mentorship list, we have included Chris Douglas, a PMC member from
>>>> Hadoop
>>>> and ASF member to help guide the community. Chris M. and Chris D. have
>>>> guided a number of projects through the Incubator over the years. The
>>>> other mentor includes Paul Ramirez, who has experience with the
>>>> Incubator
>>>> -- he was a mentor for Apache Any23, and also was  one of the PPMC
>>>> members
>>>> and eventual mentor for Apache SIS. The 70% of proposed initial
>>>> committers
>>>> that aren't as familiar with Apache have a broad range of experience
>>>>in
>>>> other open source projects, and have a deep respect and affinity for
>>>>the
>>>> foundation and the work that gets done here. The more experience ASF
>>>> mentors and project members will help to guide them.
>>>> === Community ===
>>>> There is an existing, established community of developers and users of
>>>> this projet. This includes established communities including the
>>>> Coordinated Regional Downscaling Experiment, the U.S. National Climate
>>>> Assessment (NCA), the North American Regional Climate Change
>>>>Assessment
>>>> Program (NARCCAP), and more. The Coordinated Regional Climate
>>>> Downscaling
>>>> Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is
>>>>a
>>>> world wide effort of coordination of regional climate downscaling
>>>>(RCD)
>>>> experiments driven by the World Climate Research Program (WRCP,
>>>> http://www.wcrp-climate.org/index.shtml). Recently, a large number of
>>>> RCD
>>>> projects have been carried out on a large parts of the world. To
>>>> maximize
>>>> the benefits of these research activities the WCRP designed a
>>>>framework
>>>> (Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data
>>>>sets
>>>> of
>>>> RCD-based information for the recent historical past and 21st century
>>>> projections, covering the majority of populated land regions on the
>>>> globe".  CORDEX defined different control domains (up to 10,
>>>> http://cordex.dmi.dk/joomla/) for almost all the populated regions of
>>>> the
>>>> world in a way to standardize the experiments and make them
>>>>comparable.
>>>> A
>>>> key region focused on Africa was also designated as the top priority
>>>>by
>>>> WRCP. CORDEX also provides a a series of conventions and list of
>>>> variables
>>>> that have to be followed by any project that wants to contribute to
>>>>the
>>>> experiment. Each CORDEX region has a coordinator and regional and
>>>> international periodic meetings are scheduled in a way to ensure the
>>>> global well being. NARCCAP is the U.S. contribution to CORDEX. From
>>>>the
>>>> [[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
>>>> National Climate Assessment]] site, work is "being conducted under the
>>>> auspices of the Global Change Research Act of 1990. The GCRA requires
>>>>a
>>>> report to the President and the Congress every four years that
>>>> integrates,
>>>> evaluates, and interprets the findings of the U.S. Global Change
>>>> Research
>>>> Program (USGCRP); analyzes the effects of global change on the natural
>>>> environment, agriculture, energy production and use, land and water
>>>> resources, transportation, human health and welfare, human social
>>>> systems,
>>>> and biological diversity; and analyzes current trends in global
>>>>change,
>>>> both human-induced and natural, and projects major trends for the
>>>> subsequent 25 to 100 years."
>>>> 
>>>> Apache Open Climate Workbench will support all of these communities
>>>> above,
>>>> with an eye towards being a general purpose climate evaluation toolkit
>>>> for
>>>> model output and remote sensing data.
>>>> 
>>>> === Core Developers ===
>>>> The initial set of developers comes from various NASA centers (JPL,
>>>>and
>>>> Goddard Space Flight Center), NASA HQ, various  Universities
>>>> participating
>>>> in CORDEX (Cape Town, University of New South Wales), the Indian
>>>> Institute
>>>> of Tropical Meteorology, the Free Univ. Berlin), the University of
>>>> California Los Angeles, and Howard University. As mentioned previously
>>>> several of our developers are Apache veterans and understand how it
>>>> works
>>>> around here and for those that don't, they will have great mentorship.
>>>> 
>>>> === Alignment ===
>>>> Our proposed effort aligns with the U.S. National Climate Assessment,
>>>> the
>>>> CORDEX effort, other efforts, including the Earth System Grid
>>>> Federation,
>>>> other climate software including the Open Climate GIS toolkit, other
>>>> science portals for climate including the Climate Information Portal
>>>> (CIP)
>>>> at the University of Cape Town, and other related projects.
>>>> 
>>>> There are also a number of related Apache projects and dependencies,
>>>> that
>>>> will be mentioned in the Relationships with Other Apache products
>>>> section.
>>>> 
>>>> == Known Risks ==
>>>> === Orphaned products ===
>>>> Our project has a history of funding support from JPL, NASA
>>>> (Applications
>>>> program/ARRA, NCA, AIST), NSF (ExArch project), international
>>>>investment
>>>> from collaborators, and from other funding sources. The funding
>>>>sources
>>>> are all target future deliverables and activities, so there is little
>>>> chance this software and community will be orphaned.
>>>> 
>>>> === Inexperience with Open Source ===
>>>> All the initial developers have worked on open source before -- 30% of
>>>> the
>>>> proposed initial community are experience with the ASF, and are PMC
>>>> members and committers on ASF project including 5 ASF members. Our
>>>> mentors
>>>> are all ASF members, and we welcome any interest from additional
>>>>Apache
>>>> mentors in the effort. Those 70% of our project that aren't Apache
>>>> committers, PMC members, or members will benefit from the leadership
>>>>of
>>>> the other 30% of the project.
>>>> 
>>>> === Homogenous Developers ===
>>>> The initial developers come from a variety of backgrounds and with a
>>>> variety of needs for the proposed framework. Everyone is used to
>>>> communicating on mailing lists as the project spans timezones,
>>>> international institutions and centers of excellence for climate
>>>> science.
>>>> 
>>>> === Reliance on Salaried Developers ===
>>>> All of the proposed initial developers are paid to work on this or
>>>> related
>>>> projects, but the proposed project is not the primary task for anyone.
>>>> 
>>>> === Relationships with Other Apache Products ===
>>>> As mentioned above, RCMES and the Apache Open Climate Workbench
>>>>already
>>>> depend on Apache OODT for facade interfaces to underlying data
>>>> warehouses
>>>> for storing remote sensing data; and for metadata extraction and
>>>> transformation. The software also uses Apache Tika for this (through a
>>>> transitive dependency from OODT). In addition, we have hooks to Apache
>>>> Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
>>>> remote sensing data from MySQL and into HIVE.
>>>> 
>>>> === A Excessive Fascination with the Apache Brand ===
>>>> All of us are familiar with Apache and have a respect for its brand
>>>>and
>>>> community. Chris Mattmann is a big proponent of Apache's
>>>>sustainability
>>>> factor -- and it's ability to grow software communities, in an
>>>> institution, or funding source neutral manner. All of the community
>>>>have
>>>> an extreme respect for Apache, including those in our communities who
>>>> aren't necessarily trained computer scientists, but are Scientists
>>>>(big
>>>> "S", e.g., land, physical, Earth/Climate scientists).
>>>> 
>>>> == Documentation ==
>>>> The initial RCMES code base will come from the internal JPL Subversion
>>>> repository. The [[Regional Climate Model Evaluation System (RCMES)
>>>> project|http://rcmes.jpl.nasa.gov]] at
>>>>[[JPL|http://www.jpl.nasa.gov/]]
>>>> has documentation on the existing software, including links to funding
>>>> support, communities, and other projects. We will continue to maintain
>>>> that site at JPL, which is part of the reason for rebranding the
>>>>project
>>>> here at Apache with a new name to not interfere with the existing
>>>>RCMES
>>>> one that has a following. In addition, we hope to evolve RCMES@JPL to
>>>> have
>>>> increasing levels of dependency on Apache Open Climate Workbench, so
>>>> that
>>>> we can incrementally transition with little impact to existing
>>>> customers.
>>>> 
>>>> In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange
>>>> (CDX)]]
>>>> website also has documentation on the existing software.
>>>> 
>>>> == Initial Source ==
>>>> The project will start with seed code donated by NASA JPL via Mattmann
>>>> and
>>>> the rest of the initial committers, which consists of the Regional
>>>> Climate
>>>> Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
>>>> (CDX) software. This will include the core Python API for RCMET, the
>>>> RCMED
>>>> OODT catalog project (which stores remote sensing data to
>>>>MySQL/PostGIS,
>>>> and HIVE), and the RCMED extractors for various climate formats. The
>>>> source will also include Easy-RCMET, the Python Buildout for RCMET. In
>>>> addition, we will bring along the CDX toolkit, which includes a CDX
>>>> client
>>>> package that performs subsetting, access, regridding of climate data;
>>>> and
>>>> also includes a Python Buildout installer of its own called Uber CDX.
>>>> 
>>>> == Source and Intellectual Property Submission Plan ==
>>>> All seed code and other contributions will be handled through the
>>>>normal
>>>> Apache contribution process. Mattmann has been authorized by NASA JPL
>>>>to
>>>> lead the contribution of RCMES and CDX into the Incubator via his
>>>> existing
>>>> Apache CLA, and a Software Grant to be provided.
>>>> 
>>>> We will also contact other related efforts for possible cooperation
>>>>and
>>>> contributions.
>>>> 
>>>> == External Dependencies ==
>>>> Our project depends on a number of external libraries with various
>>>> licensing conditions. An initial list of such dependencies is shown
>>>> below.
>>>> ||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
>>>> ||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
>>>> ||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT
>>>> compat||
>>>> ||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
>>>> 
>>>> 
>>>>||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modi
>>>>fi
>>>> ed
>>>> PSF license||
>>>> ||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
>>>> ||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
>>>> ||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
>>>> 
>>>> 
>>>>||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]
>>>>||
>>>> MI
>>>> T||
>>>> 
>>>> 
>>>> == Cryptography ==
>>>> The project itself will not use cryptography, but it is possible that
>>>> some
>>>> of the external software libraries will include cryptographic code to
>>>> handle features present in various science data formats. If we need to
>>>> provide an export control statement regarding cryptographic code per
>>>> Apache policy, we will follow a similar approach by Mattmann in
>>>> [[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting
>>>> lead
>>>> this effort in Apache Tika. Mattmann is familiar with this process.
>>>> 
>>>> == Required Resources ==
>>>> Mailing lists
>>>> 
>>>> * dev@climate.incubator.apache.org
>>>> * commits@climate.incubator.apache.org
>>>> * private@climate.incubator.apache.org
>>>> 
>>>> Subversion Directory
>>>> 
>>>> * https://svn.apache.org/repos/asf/incubator/climate
>>>> 
>>>> Issue Tracking
>>>> 
>>>> * JIRA CLIMATE (CLIMATE)
>>>> 
>>>> Other Resources
>>>> 
>>>> * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
>>>> * Review Board instance - CLIMATE
>>>> * Jenkins instance - CLIMATE
>>>> 
>>>> == Initial Committers ==
>>>> ||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
>>>> ||Chris A. Mattmann ||mattmann at apache dot org
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>>> ||Cameron E. Goodale ||cgoodale at apache dot org
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>>> ||Paul Ramirez ||pramirez at apache dog org
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>>> ||Andrew F. Hart ||ahart at apache dot org
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>>> ||Jinwon Kim||jkim at atmos dot ucla dot edu
>>>> ||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
>>>> System Science and Engineering]] || no||
>>>> ||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>>>> ||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
>>>> gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no
>>>> ||
>>>> ||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>>>> ||Daniel J. Crichton||crichton at apache dot
>>>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||
>>>>yes
>>>> ||
>>>> ||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
>>>> 
>>>> 
>>>>||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.
>>>>ht
>>>> ml
>>>> |Howard University]] || no ||
>>>> ||Paul Zimdars||pzimdars at apache dot
>>>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||
>>>>yes
>>>> ||
>>>> ||Chris Jack||cjack at csag dot uct dot ac dot
>>>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>>>> ||Bruce Hewitson||hewitson at csag dot uct dot ac dot
>>>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>>>> ||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
>>>> au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
>>>> ||Jason Evans||jason dot evans at unsw dot edu dot
>>>> au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
>>>> ||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
>>>> ||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes
>>>>||
>>>> ||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
>>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes
>>>>||
>>>> ||J. Sanjay||sanjay at tropmet dot res dot in ||
>>>> [[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>>>> yes
>>>> ||
>>>> ||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
>>>> ||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]]
>>>>||
>>>> yes ||
>>>> ||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
>>>> [[http://hq.nassa.gov/|NASA HQ]] || no ||
>>>> ||Laura Carriere||laura dot carriere at nasa dot gov
>>>> ||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
>>>> ||
>>>> ||Denis Nadeau|| denis dot nadeau at nasa dot
>>>> gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] ||
>>>> no
>>>> ||
>>>> 
>>>> 
>>>> == Sponsors ==
>>>> Champion
>>>> 
>>>> * Chris Mattmann (mattmann at apache dot org)
>>>> 
>>>> Nominated Mentors
>>>> 
>>>> * Chris A. Mattmann (mattmann at apache dot org)
>>>> * Chris Douglas (cdouglas at apache dot org)
>>>> * Paul Ramirez (pramirez at apache dot org)
>>>> 
>>>> Sponsoring Entity
>>>> 
>>>> * Apache Incubator
>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by Suresh Marru <sm...@apache.org>.
On Jan 27, 2013, at 1:56 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:

> Hey Suresh,
> 
> Thanks a ton man and great to hear!
> 
> To your comment below:
> 
> On 1/27/13 5:56 AM, "Suresh Marru" <sm...@apache.org> wrote:
> 
>> Hi Chris,
>> 
>> Great proposal, I am personally looking forward to see more of these
>> federal government and international collaboration efforts adopt open
>> community process. Impressive to see this project is pulling together
>> initial developers from government and universities of US, South Africa,
>> UK, Germany and India (I will hope all of them will be able to get their
>> CLA's cleared). You are really scraping the black ice on bureaucratic
>> highways and preventing at least some reinventing wheels with valuable
>> tax payers money in multiple countries.
>> 
>> Will this system evaluate and be inclusive of the community earth system
>> modeling community?
> 
> Yep this proposal includes some of those that are directly involved in
> that community including Tsengdar Lee, as well as members of the Earth
> System Grid Federation. We will definitely be inclusive and are happy to
> have involvement.

Sounds very good. I am unable to resist my itch, so will like to jump on the band wagon as a committer. Please let me know if I can add myself to the proposal wiki. 

Thanks,
Suresh

> Thanks!
> 
> Cheers,
> Chris
> 
>> 
>> Suresh
>> 
>> On Jan 26, 2013, at 2:48 PM, "Mattmann, Chris A (388J)"
>> <ch...@jpl.nasa.gov> wrote:
>> 
>>> Hi Everyone!
>>> 
>>> I bring to the Incubator a new proposal for the Apache Open Climate
>>> Workbench (incubating) project. I've added the proposal to the Incubator
>>> wiki here:
>>> 
>>> http://wiki.apache.org/incubator/ClimateProposal
>>> 
>>> 
>>> The project is a distributed, scalable approach for the rapid comparison
>>> of remote sensing data (e.g., from NASA) with that of climate model
>>> output
>>> generated by major US and international activities including the US
>>> National Climate Assessment, the Coordinated Regional Downscaling
>>> Experiment (CORDEX), the North American Regional Climate Change
>>> Assessment
>>> Program (NARCCAP) the Intergovernmental Panel on Climate Change (IPCC).
>>> Much of the software to be donated and evolved here at Apache as a
>>> community includes core dependencies to several Apache projects (OODT,
>>> Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
>>> software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.
>>> 
>>> We welcome your feedback over the next week or so for discussion and
>>> look
>>> forward to it!
>>> 
>>> Cheers,
>>> Chris Mattmann
>>> Proposed Mentor and Champion
>>> 
>>> ---------------proposal text
>>> 
>>> = Apache Open Climate Workbench, tool for scalable comparison of remote
>>> sensing observations to climate model outputs, regionally and globally.
>>> =
>>> === Abstract ===
>>> The Apache Open Climate Workbench proposal desires to contribute an
>>> existing community of software related to the analysis and evaluation of
>>> climate models, and related to the use of remote sensing data in that
>>> process. 
>>> 
>>> Specifically, we will bring a fundamental software toolkit for analysis
>>> and evaluation of climate model output against remote sensing data. The
>>> toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
>>> Evaluation System (RCMES)]]. RCMES provides two fundamental components
>>> for
>>> the easy, intuitive comparison of climate model output against remote
>>> sensing data. The first component called RCMED (for "Regional Climate
>>> Model Evaluation Database") is a scalable cloud database that decimates
>>> remote sensing data and renalysis data related to climate using Apache
>>> OODT extractors, Apache Tika, etc. These transformations make
>>> traditionally heterogeneous upstream remote sensing data and climate
>>> model
>>> output homogeneous and unify them into a data point model of the form
>>> (lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
>>> and Longitude (lng) are in WGS84 format, but can be reformatted on the
>>> fly. time is in ISO 8601 format, a string sortable format independent of
>>> underlying store. value carries with it units, related to interpretation
>>> and height allows for different values for different atmospheric
>>> vertical
>>> levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
>>> and Apache Hive, along with hooks to PostGIS and MySQL (traditional
>>> relational databases). The second component of the system, RCMET (for
>>> "Regional Climate Model Evaluation Toolkit") provides facilities for
>>> connecting to RCMED, dynamically obtaining remote sensing data for a
>>> space/time region of interest, grabbing associated model output (that
>>> the
>>> user brings, or from the Earth System Grid Federation) of the same form,
>>> and then regridding the remote sensing data to be on the model output
>>> grid, or the model output to be on the remote sensing data grid. The
>>> regridded data spatially is then temporally regridded using techniques
>>> including seasonal cycle compositing (e.g., all summer months, all
>>> Januaries, etc.), or by daily, monthly, etc. The uniform model output
>>> and
>>> remote sensing data are then analyzed using pluggable metrics, e.g.,
>>> Probability Distribution Functions (PDFs), Root Mean Squared Error
>>> (RMSE),
>>> Bias, and other (possibly user-defined) techniques, computing an
>>> analyzed
>>> comparison or evaluation. This evaluation is then visualized by plugging
>>> in to the NCAR NCL library for producing static plots (histograms, time
>>> series, etc.)
>>> 
>>> We also have performed a great deal of work in packaging RCMES to make
>>> the
>>> system easy to deploy. We have working Virtual Machines (VMWare VMX and
>>> Virtual Box OVA compatible formats) and we also have an installer built
>>> on
>>> Python Buildout (http://buildout.org/) called "Easy RCMET" for
>>> dynamically
>>> constructing the RCMET toolkit.
>>> 
>>> RCMES is currently supporting a number of recognized climate projects of
>>> (inter-)national significance. In particular, RCMES is supporting the
>>> [[http://www.globalchange.gov/what-we-do/assessment|U.S. National
>>> Climate
>>> Assessment (NCA) activities]] on behalf of NASA's contribution to the
>>> NCA;
>>> is working with the [[http://www.narccap.ucar.edu/|North American
>>> Regional
>>> Climate Change Assessment Program (NARCCAP)]]; and is also working with
>>> the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
>>> Regional Downscaling Experiment (CORDEX)]].
>>> 
>>> === Proposal ===
>>> We propose to transition the RCMES software community, which includes
>>> developers of the RCMET and RCMED software, along with users of RCMES in
>>> the CORDEX project across a variety of academic institutions, scientists
>>> helping to improve the RCMES metrics, and visualizations, and regridding
>>> algorithms, packagers making RCMES easier to install, and scientists
>>> helping to lead some of these international projects that are already
>>> using RCMES. 
>>> 
>>> We have been working on the RCMES project since 2009 funded initially by
>>> the American Recovery and Reinvestment Act (ARRA) project out at NASA,
>>> and
>>> then branching out into other sources of support and sustainability
>>> (NASA;
>>> NSF, etc. -- see the
>>> [[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
>>> the RCMES website for a full list of supporting U.S. and international
>>> partners). 
>>> 
>>> With the existing RCMES community at Apache, we will also work to
>>> encourage other climate software projects e.g., Open Climate GIS,
>>> elements
>>> of the Earth System Grid Federation, other NASA climate projects funded
>>> under the Computational Modeling, Algorithms and Cyberinfrastructure
>>> (CMAC) to contribute to the Open Climate Workbench here at Apache.
>>> 
>>> RCMED is a Big Data project that combines several underlying Apache
>>> software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related
>>> data
>>> management software. Its primary language is Java; RCMET, on the other
>>> hand, is a Python API, associated set of classes (framework), set of
>>> Python Bottle Web services, and a PHP "Wizard"-based User Interface that
>>> leverages Apache OODT Balance.
>>> 
>>> === Background ===
>>> Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
>>> his solid experience with Apache OODT and bringing it to the ASF. Chris
>>> worked for a year to get the support of the JPL community including
>>> approvals from the Software Release authority at JPL to release the
>>> software. 
>>> 
>>> The initial code drop will include the RCMES SVN repository from JPL
>>> including prior revisions. We anticipate also including a smaller
>>> package,
>>> CDX, which contains some useful facilities for regridding, and command
>>> line tools for manipulating large datasets, and working with OPeNDAP,
>>> etc.
>>> 
>>> After the code drop, we will work with our developers, users,
>>> documentors,
>>> and other members of the team to teach those unfamiliar with the Apache
>>> way how it works around here at Apache. 30% of the community from RCMES
>>> includes those intimately familiar with Apache including 5 ASF members
>>> --
>>> the other 70% include a range of scientific code developers, climate
>>> scientists that use RCMES, program officers that will help make
>>> documentation and slides for the code, and advocate for it in the
>>> community. Their experience with Apache ranges from using various ASF
>>> products, to contributing patches to them, to not using any ASF software
>>> at all.
>>> 
>>> With this diversity, we anticipate that while everything may not just
>>> work
>>> turnkey out of the box, this represents a unique opportunity to
>>> demonstrate Apache to the international community and to show the
>>> benefits
>>> of its community and social models. That said, we also have a lot of ASF
>>> experience to make sure everyone learns the Apache way.
>>> 
>>> === Rationale ===
>>> We are bringing RCMES to Apache for a few reasons. First, we feel that
>>> it
>>> will immediately enable our collaborators across a number of
>>> institutions
>>> both nationally and internationally have the opportunity to work on a
>>> common software base, and to improve it with contributions from their
>>> own
>>> sites. Currently these are difficult to negotiate now because of varied
>>> legal and contribution frameworks -- Apache allows us to simplify this
>>> to
>>> a unified one. Second, using the ASF's world-wide mirroring system, we
>>> will be able to deliver climate software broadly to the community as we
>>> release it, rather than sneaker netting the software around or
>>> establishing our own point release infrastructure.
>>> 
>>> Bringing this project to Apache also immediately thrusts the ASF into
>>> the
>>> thriving ecosystem of the Coordinated Regional Downscaling Experiment
>>> (CORDEX), the US National Cimate Assessment, the North American Regional
>>> Climate Change Assessment Program (the US contribution to CORDEX) and
>>> into
>>> relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
>>> assessment activities at a number of different institutions. We also
>>> seek
>>> to help lead and encourage du jour standard development rather than top
>>> down level dictating of standards for climate software and the ASF will
>>> provide us a means for that.
>>> 
>>> === Initial Goals ===
>>> The initial goals of the proposed project are:
>>> 
>>> * Stand up a sustaining Apache-based community around the JPL RCMES
>>> codebase.
>>> * Active relationships and possible cooperation with related projects
>>> and
>>> communities, including end user and scientific communities, CORDEX,
>>> NARCCAP, US NCA, IPCC, ESG, etc.
>>> * Active relationships and possible cooperation with existing Apache
>>> communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
>>> * Initial Apache release.
>>> * Leverage Apache Open Climate Workbench in climate activities at NASA,
>>> in the international community as mentioned above, and beyond.
>>> * Vetting all software licenses and making sure IP is clear (software
>>> grant from JPL forthcoming).
>>> 
>>> == Current Status ==
>>> === Meritocracy ===
>>> 30% of the proposed initial committers are familiar with the meritocracy
>>> principles of Apache. As stated above this includes 5 ASF members. Of
>>> the
>>> mentorship list, we have included Chris Douglas, a PMC member from
>>> Hadoop
>>> and ASF member to help guide the community. Chris M. and Chris D. have
>>> guided a number of projects through the Incubator over the years. The
>>> other mentor includes Paul Ramirez, who has experience with the
>>> Incubator
>>> -- he was a mentor for Apache Any23, and also was  one of the PPMC
>>> members
>>> and eventual mentor for Apache SIS. The 70% of proposed initial
>>> committers
>>> that aren't as familiar with Apache have a broad range of experience in
>>> other open source projects, and have a deep respect and affinity for the
>>> foundation and the work that gets done here. The more experience ASF
>>> mentors and project members will help to guide them.
>>> === Community ===
>>> There is an existing, established community of developers and users of
>>> this projet. This includes established communities including the
>>> Coordinated Regional Downscaling Experiment, the U.S. National Climate
>>> Assessment (NCA), the North American Regional Climate Change Assessment
>>> Program (NARCCAP), and more. The Coordinated Regional Climate
>>> Downscaling
>>> Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
>>> world wide effort of coordination of regional climate downscaling (RCD)
>>> experiments driven by the World Climate Research Program (WRCP,
>>> http://www.wcrp-climate.org/index.shtml). Recently, a large number of
>>> RCD
>>> projects have been carried out on a large parts of the world. To
>>> maximize
>>> the benefits of these research activities the WCRP designed a framework
>>> (Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets
>>> of
>>> RCD-based information for the recent historical past and 21st century
>>> projections, covering the majority of populated land regions on the
>>> globe".  CORDEX defined different control domains (up to 10,
>>> http://cordex.dmi.dk/joomla/) for almost all the populated regions of
>>> the
>>> world in a way to standardize the experiments and make them comparable.
>>> A
>>> key region focused on Africa was also designated as the top priority by
>>> WRCP. CORDEX also provides a a series of conventions and list of
>>> variables
>>> that have to be followed by any project that wants to contribute to the
>>> experiment. Each CORDEX region has a coordinator and regional and
>>> international periodic meetings are scheduled in a way to ensure the
>>> global well being. NARCCAP is the U.S. contribution to CORDEX. From the
>>> [[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
>>> National Climate Assessment]] site, work is "being conducted under the
>>> auspices of the Global Change Research Act of 1990. The GCRA requires a
>>> report to the President and the Congress every four years that
>>> integrates,
>>> evaluates, and interprets the findings of the U.S. Global Change
>>> Research
>>> Program (USGCRP); analyzes the effects of global change on the natural
>>> environment, agriculture, energy production and use, land and water
>>> resources, transportation, human health and welfare, human social
>>> systems,
>>> and biological diversity; and analyzes current trends in global change,
>>> both human-induced and natural, and projects major trends for the
>>> subsequent 25 to 100 years."
>>> 
>>> Apache Open Climate Workbench will support all of these communities
>>> above,
>>> with an eye towards being a general purpose climate evaluation toolkit
>>> for
>>> model output and remote sensing data.
>>> 
>>> === Core Developers ===
>>> The initial set of developers comes from various NASA centers (JPL, and
>>> Goddard Space Flight Center), NASA HQ, various  Universities
>>> participating
>>> in CORDEX (Cape Town, University of New South Wales), the Indian
>>> Institute
>>> of Tropical Meteorology, the Free Univ. Berlin), the University of
>>> California Los Angeles, and Howard University. As mentioned previously
>>> several of our developers are Apache veterans and understand how it
>>> works
>>> around here and for those that don't, they will have great mentorship.
>>> 
>>> === Alignment ===
>>> Our proposed effort aligns with the U.S. National Climate Assessment,
>>> the
>>> CORDEX effort, other efforts, including the Earth System Grid
>>> Federation,
>>> other climate software including the Open Climate GIS toolkit, other
>>> science portals for climate including the Climate Information Portal
>>> (CIP)
>>> at the University of Cape Town, and other related projects.
>>> 
>>> There are also a number of related Apache projects and dependencies,
>>> that
>>> will be mentioned in the Relationships with Other Apache products
>>> section.
>>> 
>>> == Known Risks ==
>>> === Orphaned products ===
>>> Our project has a history of funding support from JPL, NASA
>>> (Applications
>>> program/ARRA, NCA, AIST), NSF (ExArch project), international investment
>>> from collaborators, and from other funding sources. The funding sources
>>> are all target future deliverables and activities, so there is little
>>> chance this software and community will be orphaned.
>>> 
>>> === Inexperience with Open Source ===
>>> All the initial developers have worked on open source before -- 30% of
>>> the
>>> proposed initial community are experience with the ASF, and are PMC
>>> members and committers on ASF project including 5 ASF members. Our
>>> mentors
>>> are all ASF members, and we welcome any interest from additional Apache
>>> mentors in the effort. Those 70% of our project that aren't Apache
>>> committers, PMC members, or members will benefit from the leadership of
>>> the other 30% of the project.
>>> 
>>> === Homogenous Developers ===
>>> The initial developers come from a variety of backgrounds and with a
>>> variety of needs for the proposed framework. Everyone is used to
>>> communicating on mailing lists as the project spans timezones,
>>> international institutions and centers of excellence for climate
>>> science.
>>> 
>>> === Reliance on Salaried Developers ===
>>> All of the proposed initial developers are paid to work on this or
>>> related
>>> projects, but the proposed project is not the primary task for anyone.
>>> 
>>> === Relationships with Other Apache Products ===
>>> As mentioned above, RCMES and the Apache Open Climate Workbench already
>>> depend on Apache OODT for facade interfaces to underlying data
>>> warehouses
>>> for storing remote sensing data; and for metadata extraction and
>>> transformation. The software also uses Apache Tika for this (through a
>>> transitive dependency from OODT). In addition, we have hooks to Apache
>>> Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
>>> remote sensing data from MySQL and into HIVE.
>>> 
>>> === A Excessive Fascination with the Apache Brand ===
>>> All of us are familiar with Apache and have a respect for its brand and
>>> community. Chris Mattmann is a big proponent of Apache's sustainability
>>> factor -- and it's ability to grow software communities, in an
>>> institution, or funding source neutral manner. All of the community have
>>> an extreme respect for Apache, including those in our communities who
>>> aren't necessarily trained computer scientists, but are Scientists (big
>>> "S", e.g., land, physical, Earth/Climate scientists).
>>> 
>>> == Documentation ==
>>> The initial RCMES code base will come from the internal JPL Subversion
>>> repository. The [[Regional Climate Model Evaluation System (RCMES)
>>> project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
>>> has documentation on the existing software, including links to funding
>>> support, communities, and other projects. We will continue to maintain
>>> that site at JPL, which is part of the reason for rebranding the project
>>> here at Apache with a new name to not interfere with the existing RCMES
>>> one that has a following. In addition, we hope to evolve RCMES@JPL to
>>> have
>>> increasing levels of dependency on Apache Open Climate Workbench, so
>>> that
>>> we can incrementally transition with little impact to existing
>>> customers.
>>> 
>>> In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange
>>> (CDX)]]
>>> website also has documentation on the existing software.
>>> 
>>> == Initial Source ==
>>> The project will start with seed code donated by NASA JPL via Mattmann
>>> and
>>> the rest of the initial committers, which consists of the Regional
>>> Climate
>>> Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
>>> (CDX) software. This will include the core Python API for RCMET, the
>>> RCMED
>>> OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
>>> and HIVE), and the RCMED extractors for various climate formats. The
>>> source will also include Easy-RCMET, the Python Buildout for RCMET. In
>>> addition, we will bring along the CDX toolkit, which includes a CDX
>>> client
>>> package that performs subsetting, access, regridding of climate data;
>>> and
>>> also includes a Python Buildout installer of its own called Uber CDX.
>>> 
>>> == Source and Intellectual Property Submission Plan ==
>>> All seed code and other contributions will be handled through the normal
>>> Apache contribution process. Mattmann has been authorized by NASA JPL to
>>> lead the contribution of RCMES and CDX into the Incubator via his
>>> existing
>>> Apache CLA, and a Software Grant to be provided.
>>> 
>>> We will also contact other related efforts for possible cooperation and
>>> contributions.
>>> 
>>> == External Dependencies ==
>>> Our project depends on a number of external libraries with various
>>> licensing conditions. An initial list of such dependencies is shown
>>> below.
>>> ||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
>>> ||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
>>> ||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT
>>> compat||
>>> ||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
>>> 
>>> ||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modifi
>>> ed
>>> PSF license||
>>> ||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
>>> ||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
>>> ||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
>>> 
>>> ||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||
>>> MI
>>> T||
>>> 
>>> 
>>> == Cryptography ==
>>> The project itself will not use cryptography, but it is possible that
>>> some
>>> of the external software libraries will include cryptographic code to
>>> handle features present in various science data formats. If we need to
>>> provide an export control statement regarding cryptographic code per
>>> Apache policy, we will follow a similar approach by Mattmann in
>>> [[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting
>>> lead
>>> this effort in Apache Tika. Mattmann is familiar with this process.
>>> 
>>> == Required Resources ==
>>> Mailing lists
>>> 
>>> * dev@climate.incubator.apache.org
>>> * commits@climate.incubator.apache.org
>>> * private@climate.incubator.apache.org
>>> 
>>> Subversion Directory
>>> 
>>> * https://svn.apache.org/repos/asf/incubator/climate
>>> 
>>> Issue Tracking
>>> 
>>> * JIRA CLIMATE (CLIMATE)
>>> 
>>> Other Resources
>>> 
>>> * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
>>> * Review Board instance - CLIMATE
>>> * Jenkins instance - CLIMATE
>>> 
>>> == Initial Committers ==
>>> ||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
>>> ||Chris A. Mattmann ||mattmann at apache dot org
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>> ||Cameron E. Goodale ||cgoodale at apache dot org
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>> ||Paul Ramirez ||pramirez at apache dog org
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>> ||Andrew F. Hart ||ahart at apache dot org
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>>> ||Jinwon Kim||jkim at atmos dot ucla dot edu
>>> ||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
>>> System Science and Engineering]] || no||
>>> ||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>>> ||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
>>> gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no
>>> ||
>>> ||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>>> ||Daniel J. Crichton||crichton at apache dot
>>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes
>>> ||
>>> ||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
>>> 
>>> ||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.ht
>>> ml
>>> |Howard University]] || no ||
>>> ||Paul Zimdars||pzimdars at apache dot
>>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes
>>> ||
>>> ||Chris Jack||cjack at csag dot uct dot ac dot
>>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>>> ||Bruce Hewitson||hewitson at csag dot uct dot ac dot
>>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>>> ||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
>>> au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
>>> ||Jason Evans||jason dot evans at unsw dot edu dot
>>> au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
>>> ||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
>>> ||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
>>> ||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
>>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
>>> ||J. Sanjay||sanjay at tropmet dot res dot in ||
>>> [[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>>> yes
>>> ||
>>> ||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
>>> ||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>>> yes ||
>>> ||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
>>> [[http://hq.nassa.gov/|NASA HQ]] || no ||
>>> ||Laura Carriere||laura dot carriere at nasa dot gov
>>> ||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
>>> ||
>>> ||Denis Nadeau|| denis dot nadeau at nasa dot
>>> gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] ||
>>> no
>>> ||
>>> 
>>> 
>>> == Sponsors ==
>>> Champion
>>> 
>>> * Chris Mattmann (mattmann at apache dot org)
>>> 
>>> Nominated Mentors
>>> 
>>> * Chris A. Mattmann (mattmann at apache dot org)
>>> * Chris Douglas (cdouglas at apache dot org)
>>> * Paul Ramirez (pramirez at apache dot org)
>>> 
>>> Sponsoring Entity
>>> 
>>> * Apache Incubator
>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Suresh,

Thanks a ton man and great to hear!

To your comment below:

On 1/27/13 5:56 AM, "Suresh Marru" <sm...@apache.org> wrote:

>Hi Chris,
>
>Great proposal, I am personally looking forward to see more of these
>federal government and international collaboration efforts adopt open
>community process. Impressive to see this project is pulling together
>initial developers from government and universities of US, South Africa,
>UK, Germany and India (I will hope all of them will be able to get their
>CLA's cleared). You are really scraping the black ice on bureaucratic
>highways and preventing at least some reinventing wheels with valuable
>tax payers money in multiple countries.
>
>Will this system evaluate and be inclusive of the community earth system
>modeling community?

Yep this proposal includes some of those that are directly involved in
that community including Tsengdar Lee, as well as members of the Earth
System Grid Federation. We will definitely be inclusive and are happy to
have involvement.

Thanks!

Cheers,
Chris

>
>Suresh
>
>On Jan 26, 2013, at 2:48 PM, "Mattmann, Chris A (388J)"
><ch...@jpl.nasa.gov> wrote:
>
>> Hi Everyone!
>> 
>> I bring to the Incubator a new proposal for the Apache Open Climate
>> Workbench (incubating) project. I've added the proposal to the Incubator
>> wiki here:
>> 
>> http://wiki.apache.org/incubator/ClimateProposal
>> 
>> 
>> The project is a distributed, scalable approach for the rapid comparison
>> of remote sensing data (e.g., from NASA) with that of climate model
>>output
>> generated by major US and international activities including the US
>> National Climate Assessment, the Coordinated Regional Downscaling
>> Experiment (CORDEX), the North American Regional Climate Change
>>Assessment
>> Program (NARCCAP) the Intergovernmental Panel on Climate Change (IPCC).
>> Much of the software to be donated and evolved here at Apache as a
>> community includes core dependencies to several Apache projects (OODT,
>> Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
>> software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.
>> 
>> We welcome your feedback over the next week or so for discussion and
>>look
>> forward to it!
>> 
>> Cheers,
>> Chris Mattmann
>> Proposed Mentor and Champion
>> 
>> ---------------proposal text
>> 
>> = Apache Open Climate Workbench, tool for scalable comparison of remote
>> sensing observations to climate model outputs, regionally and globally.
>>=
>> === Abstract ===
>> The Apache Open Climate Workbench proposal desires to contribute an
>> existing community of software related to the analysis and evaluation of
>> climate models, and related to the use of remote sensing data in that
>> process. 
>> 
>> Specifically, we will bring a fundamental software toolkit for analysis
>> and evaluation of climate model output against remote sensing data. The
>> toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
>> Evaluation System (RCMES)]]. RCMES provides two fundamental components
>>for
>> the easy, intuitive comparison of climate model output against remote
>> sensing data. The first component called RCMED (for "Regional Climate
>> Model Evaluation Database") is a scalable cloud database that decimates
>> remote sensing data and renalysis data related to climate using Apache
>> OODT extractors, Apache Tika, etc. These transformations make
>> traditionally heterogeneous upstream remote sensing data and climate
>>model
>> output homogeneous and unify them into a data point model of the form
>> (lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
>> and Longitude (lng) are in WGS84 format, but can be reformatted on the
>> fly. time is in ISO 8601 format, a string sortable format independent of
>> underlying store. value carries with it units, related to interpretation
>> and height allows for different values for different atmospheric
>>vertical
>> levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
>> and Apache Hive, along with hooks to PostGIS and MySQL (traditional
>> relational databases). The second component of the system, RCMET (for
>> "Regional Climate Model Evaluation Toolkit") provides facilities for
>> connecting to RCMED, dynamically obtaining remote sensing data for a
>> space/time region of interest, grabbing associated model output (that
>>the
>> user brings, or from the Earth System Grid Federation) of the same form,
>> and then regridding the remote sensing data to be on the model output
>> grid, or the model output to be on the remote sensing data grid. The
>> regridded data spatially is then temporally regridded using techniques
>> including seasonal cycle compositing (e.g., all summer months, all
>> Januaries, etc.), or by daily, monthly, etc. The uniform model output
>>and
>> remote sensing data are then analyzed using pluggable metrics, e.g.,
>> Probability Distribution Functions (PDFs), Root Mean Squared Error
>>(RMSE),
>> Bias, and other (possibly user-defined) techniques, computing an
>>analyzed
>> comparison or evaluation. This evaluation is then visualized by plugging
>> in to the NCAR NCL library for producing static plots (histograms, time
>> series, etc.)
>> 
>> We also have performed a great deal of work in packaging RCMES to make
>>the
>> system easy to deploy. We have working Virtual Machines (VMWare VMX and
>> Virtual Box OVA compatible formats) and we also have an installer built
>>on
>> Python Buildout (http://buildout.org/) called "Easy RCMET" for
>>dynamically
>> constructing the RCMET toolkit.
>> 
>> RCMES is currently supporting a number of recognized climate projects of
>> (inter-)national significance. In particular, RCMES is supporting the
>> [[http://www.globalchange.gov/what-we-do/assessment|U.S. National
>>Climate
>> Assessment (NCA) activities]] on behalf of NASA's contribution to the
>>NCA;
>> is working with the [[http://www.narccap.ucar.edu/|North American
>>Regional
>> Climate Change Assessment Program (NARCCAP)]]; and is also working with
>> the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
>> Regional Downscaling Experiment (CORDEX)]].
>> 
>> === Proposal ===
>> We propose to transition the RCMES software community, which includes
>> developers of the RCMET and RCMED software, along with users of RCMES in
>> the CORDEX project across a variety of academic institutions, scientists
>> helping to improve the RCMES metrics, and visualizations, and regridding
>> algorithms, packagers making RCMES easier to install, and scientists
>> helping to lead some of these international projects that are already
>> using RCMES. 
>> 
>> We have been working on the RCMES project since 2009 funded initially by
>> the American Recovery and Reinvestment Act (ARRA) project out at NASA,
>>and
>> then branching out into other sources of support and sustainability
>>(NASA;
>> NSF, etc. -- see the
>> [[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
>> the RCMES website for a full list of supporting U.S. and international
>> partners). 
>> 
>> With the existing RCMES community at Apache, we will also work to
>> encourage other climate software projects e.g., Open Climate GIS,
>>elements
>> of the Earth System Grid Federation, other NASA climate projects funded
>> under the Computational Modeling, Algorithms and Cyberinfrastructure
>> (CMAC) to contribute to the Open Climate Workbench here at Apache.
>> 
>> RCMED is a Big Data project that combines several underlying Apache
>> software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related
>>data
>> management software. Its primary language is Java; RCMET, on the other
>> hand, is a Python API, associated set of classes (framework), set of
>> Python Bottle Web services, and a PHP "Wizard"-based User Interface that
>> leverages Apache OODT Balance.
>> 
>> === Background ===
>> Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
>> his solid experience with Apache OODT and bringing it to the ASF. Chris
>> worked for a year to get the support of the JPL community including
>> approvals from the Software Release authority at JPL to release the
>> software. 
>> 
>> The initial code drop will include the RCMES SVN repository from JPL
>> including prior revisions. We anticipate also including a smaller
>>package,
>> CDX, which contains some useful facilities for regridding, and command
>> line tools for manipulating large datasets, and working with OPeNDAP,
>>etc.
>> 
>> After the code drop, we will work with our developers, users,
>>documentors,
>> and other members of the team to teach those unfamiliar with the Apache
>> way how it works around here at Apache. 30% of the community from RCMES
>> includes those intimately familiar with Apache including 5 ASF members
>>--
>> the other 70% include a range of scientific code developers, climate
>> scientists that use RCMES, program officers that will help make
>> documentation and slides for the code, and advocate for it in the
>> community. Their experience with Apache ranges from using various ASF
>> products, to contributing patches to them, to not using any ASF software
>> at all.
>> 
>> With this diversity, we anticipate that while everything may not just
>>work
>> turnkey out of the box, this represents a unique opportunity to
>> demonstrate Apache to the international community and to show the
>>benefits
>> of its community and social models. That said, we also have a lot of ASF
>> experience to make sure everyone learns the Apache way.
>> 
>> === Rationale ===
>> We are bringing RCMES to Apache for a few reasons. First, we feel that
>>it
>> will immediately enable our collaborators across a number of
>>institutions
>> both nationally and internationally have the opportunity to work on a
>> common software base, and to improve it with contributions from their
>>own
>> sites. Currently these are difficult to negotiate now because of varied
>> legal and contribution frameworks -- Apache allows us to simplify this
>>to
>> a unified one. Second, using the ASF's world-wide mirroring system, we
>> will be able to deliver climate software broadly to the community as we
>> release it, rather than sneaker netting the software around or
>> establishing our own point release infrastructure.
>> 
>> Bringing this project to Apache also immediately thrusts the ASF into
>>the
>> thriving ecosystem of the Coordinated Regional Downscaling Experiment
>> (CORDEX), the US National Cimate Assessment, the North American Regional
>> Climate Change Assessment Program (the US contribution to CORDEX) and
>>into
>> relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
>> assessment activities at a number of different institutions. We also
>>seek
>> to help lead and encourage du jour standard development rather than top
>> down level dictating of standards for climate software and the ASF will
>> provide us a means for that.
>> 
>> === Initial Goals ===
>> The initial goals of the proposed project are:
>> 
>> * Stand up a sustaining Apache-based community around the JPL RCMES
>> codebase.
>> * Active relationships and possible cooperation with related projects
>>and
>> communities, including end user and scientific communities, CORDEX,
>> NARCCAP, US NCA, IPCC, ESG, etc.
>> * Active relationships and possible cooperation with existing Apache
>> communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
>> * Initial Apache release.
>> * Leverage Apache Open Climate Workbench in climate activities at NASA,
>> in the international community as mentioned above, and beyond.
>> * Vetting all software licenses and making sure IP is clear (software
>> grant from JPL forthcoming).
>> 
>> == Current Status ==
>> === Meritocracy ===
>> 30% of the proposed initial committers are familiar with the meritocracy
>> principles of Apache. As stated above this includes 5 ASF members. Of
>>the
>> mentorship list, we have included Chris Douglas, a PMC member from
>>Hadoop
>> and ASF member to help guide the community. Chris M. and Chris D. have
>> guided a number of projects through the Incubator over the years. The
>> other mentor includes Paul Ramirez, who has experience with the
>>Incubator
>> -- he was a mentor for Apache Any23, and also was  one of the PPMC
>>members
>> and eventual mentor for Apache SIS. The 70% of proposed initial
>>committers
>> that aren't as familiar with Apache have a broad range of experience in
>> other open source projects, and have a deep respect and affinity for the
>> foundation and the work that gets done here. The more experience ASF
>> mentors and project members will help to guide them.
>> === Community ===
>> There is an existing, established community of developers and users of
>> this projet. This includes established communities including the
>> Coordinated Regional Downscaling Experiment, the U.S. National Climate
>> Assessment (NCA), the North American Regional Climate Change Assessment
>> Program (NARCCAP), and more. The Coordinated Regional Climate
>>Downscaling
>> Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
>> world wide effort of coordination of regional climate downscaling (RCD)
>> experiments driven by the World Climate Research Program (WRCP,
>> http://www.wcrp-climate.org/index.shtml). Recently, a large number of
>>RCD
>> projects have been carried out on a large parts of the world. To
>>maximize
>> the benefits of these research activities the WCRP designed a framework
>> (Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets
>>of
>> RCD-based information for the recent historical past and 21st century
>> projections, covering the majority of populated land regions on the
>> globe".  CORDEX defined different control domains (up to 10,
>> http://cordex.dmi.dk/joomla/) for almost all the populated regions of
>>the
>> world in a way to standardize the experiments and make them comparable.
>>A
>> key region focused on Africa was also designated as the top priority by
>> WRCP. CORDEX also provides a a series of conventions and list of
>>variables
>> that have to be followed by any project that wants to contribute to the
>> experiment. Each CORDEX region has a coordinator and regional and
>> international periodic meetings are scheduled in a way to ensure the
>> global well being. NARCCAP is the U.S. contribution to CORDEX. From the
>> [[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
>> National Climate Assessment]] site, work is "being conducted under the
>> auspices of the Global Change Research Act of 1990. The GCRA requires a
>> report to the President and the Congress every four years that
>>integrates,
>> evaluates, and interprets the findings of the U.S. Global Change
>>Research
>> Program (USGCRP); analyzes the effects of global change on the natural
>> environment, agriculture, energy production and use, land and water
>> resources, transportation, human health and welfare, human social
>>systems,
>> and biological diversity; and analyzes current trends in global change,
>> both human-induced and natural, and projects major trends for the
>> subsequent 25 to 100 years."
>> 
>> Apache Open Climate Workbench will support all of these communities
>>above,
>> with an eye towards being a general purpose climate evaluation toolkit
>>for
>> model output and remote sensing data.
>> 
>> === Core Developers ===
>> The initial set of developers comes from various NASA centers (JPL, and
>> Goddard Space Flight Center), NASA HQ, various  Universities
>>participating
>> in CORDEX (Cape Town, University of New South Wales), the Indian
>>Institute
>> of Tropical Meteorology, the Free Univ. Berlin), the University of
>> California Los Angeles, and Howard University. As mentioned previously
>> several of our developers are Apache veterans and understand how it
>>works
>> around here and for those that don't, they will have great mentorship.
>> 
>> === Alignment ===
>> Our proposed effort aligns with the U.S. National Climate Assessment,
>>the
>> CORDEX effort, other efforts, including the Earth System Grid
>>Federation,
>> other climate software including the Open Climate GIS toolkit, other
>> science portals for climate including the Climate Information Portal
>>(CIP)
>> at the University of Cape Town, and other related projects.
>> 
>> There are also a number of related Apache projects and dependencies,
>>that
>> will be mentioned in the Relationships with Other Apache products
>>section.
>> 
>> == Known Risks ==
>> === Orphaned products ===
>> Our project has a history of funding support from JPL, NASA
>>(Applications
>> program/ARRA, NCA, AIST), NSF (ExArch project), international investment
>> from collaborators, and from other funding sources. The funding sources
>> are all target future deliverables and activities, so there is little
>> chance this software and community will be orphaned.
>> 
>> === Inexperience with Open Source ===
>> All the initial developers have worked on open source before -- 30% of
>>the
>> proposed initial community are experience with the ASF, and are PMC
>> members and committers on ASF project including 5 ASF members. Our
>>mentors
>> are all ASF members, and we welcome any interest from additional Apache
>> mentors in the effort. Those 70% of our project that aren't Apache
>> committers, PMC members, or members will benefit from the leadership of
>> the other 30% of the project.
>> 
>> === Homogenous Developers ===
>> The initial developers come from a variety of backgrounds and with a
>> variety of needs for the proposed framework. Everyone is used to
>> communicating on mailing lists as the project spans timezones,
>> international institutions and centers of excellence for climate
>>science.
>> 
>> === Reliance on Salaried Developers ===
>> All of the proposed initial developers are paid to work on this or
>>related
>> projects, but the proposed project is not the primary task for anyone.
>> 
>> === Relationships with Other Apache Products ===
>> As mentioned above, RCMES and the Apache Open Climate Workbench already
>> depend on Apache OODT for facade interfaces to underlying data
>>warehouses
>> for storing remote sensing data; and for metadata extraction and
>> transformation. The software also uses Apache Tika for this (through a
>> transitive dependency from OODT). In addition, we have hooks to Apache
>> Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
>> remote sensing data from MySQL and into HIVE.
>> 
>> === A Excessive Fascination with the Apache Brand ===
>> All of us are familiar with Apache and have a respect for its brand and
>> community. Chris Mattmann is a big proponent of Apache's sustainability
>> factor -- and it's ability to grow software communities, in an
>> institution, or funding source neutral manner. All of the community have
>> an extreme respect for Apache, including those in our communities who
>> aren't necessarily trained computer scientists, but are Scientists (big
>> "S", e.g., land, physical, Earth/Climate scientists).
>> 
>> == Documentation ==
>> The initial RCMES code base will come from the internal JPL Subversion
>> repository. The [[Regional Climate Model Evaluation System (RCMES)
>> project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
>> has documentation on the existing software, including links to funding
>> support, communities, and other projects. We will continue to maintain
>> that site at JPL, which is part of the reason for rebranding the project
>> here at Apache with a new name to not interfere with the existing RCMES
>> one that has a following. In addition, we hope to evolve RCMES@JPL to
>>have
>> increasing levels of dependency on Apache Open Climate Workbench, so
>>that
>> we can incrementally transition with little impact to existing
>>customers.
>> 
>> In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange
>>(CDX)]]
>> website also has documentation on the existing software.
>> 
>> == Initial Source ==
>> The project will start with seed code donated by NASA JPL via Mattmann
>>and
>> the rest of the initial committers, which consists of the Regional
>>Climate
>> Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
>> (CDX) software. This will include the core Python API for RCMET, the
>>RCMED
>> OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
>> and HIVE), and the RCMED extractors for various climate formats. The
>> source will also include Easy-RCMET, the Python Buildout for RCMET. In
>> addition, we will bring along the CDX toolkit, which includes a CDX
>>client
>> package that performs subsetting, access, regridding of climate data;
>>and
>> also includes a Python Buildout installer of its own called Uber CDX.
>> 
>> == Source and Intellectual Property Submission Plan ==
>> All seed code and other contributions will be handled through the normal
>> Apache contribution process. Mattmann has been authorized by NASA JPL to
>> lead the contribution of RCMES and CDX into the Incubator via his
>>existing
>> Apache CLA, and a Software Grant to be provided.
>> 
>> We will also contact other related efforts for possible cooperation and
>> contributions.
>> 
>> == External Dependencies ==
>> Our project depends on a number of external libraries with various
>> licensing conditions. An initial list of such dependencies is shown
>>below.
>> ||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
>> ||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
>> ||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT
>>compat||
>> ||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
>> 
>>||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modifi
>>ed
>> PSF license||
>> ||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
>> ||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
>> ||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
>> 
>>||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||
>>MI
>> T||
>> 
>> 
>> == Cryptography ==
>> The project itself will not use cryptography, but it is possible that
>>some
>> of the external software libraries will include cryptographic code to
>> handle features present in various science data formats. If we need to
>> provide an export control statement regarding cryptographic code per
>> Apache policy, we will follow a similar approach by Mattmann in
>> [[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting
>>lead
>> this effort in Apache Tika. Mattmann is familiar with this process.
>> 
>> == Required Resources ==
>> Mailing lists
>> 
>> * dev@climate.incubator.apache.org
>> * commits@climate.incubator.apache.org
>> * private@climate.incubator.apache.org
>> 
>> Subversion Directory
>> 
>> * https://svn.apache.org/repos/asf/incubator/climate
>> 
>> Issue Tracking
>> 
>> * JIRA CLIMATE (CLIMATE)
>> 
>> Other Resources
>> 
>> * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
>> * Review Board instance - CLIMATE
>> * Jenkins instance - CLIMATE
>> 
>> == Initial Committers ==
>> ||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
>> ||Chris A. Mattmann ||mattmann at apache dot org
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>> ||Cameron E. Goodale ||cgoodale at apache dot org
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>> ||Paul Ramirez ||pramirez at apache dog org
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>> ||Andrew F. Hart ||ahart at apache dot org
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>> ||Jinwon Kim||jkim at atmos dot ucla dot edu
>> ||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
>> System Science and Engineering]] || no||
>> ||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>> ||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
>> gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no
>>||
>> ||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>> ||Daniel J. Crichton||crichton at apache dot
>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes
>>||
>> ||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
>> 
>>||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.ht
>>ml
>> |Howard University]] || no ||
>> ||Paul Zimdars||pzimdars at apache dot
>> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes
>>||
>> ||Chris Jack||cjack at csag dot uct dot ac dot
>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>> ||Bruce Hewitson||hewitson at csag dot uct dot ac dot
>> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>> ||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
>> au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
>> ||Jason Evans||jason dot evans at unsw dot edu dot
>> au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
>> ||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
>> ||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
>> ||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
>> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
>> ||J. Sanjay||sanjay at tropmet dot res dot in ||
>> [[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>>yes
>> ||
>> ||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
>> ||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>> yes ||
>> ||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
>> [[http://hq.nassa.gov/|NASA HQ]] || no ||
>> ||Laura Carriere||laura dot carriere at nasa dot gov
>> ||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
>>||
>> ||Denis Nadeau|| denis dot nadeau at nasa dot
>> gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] ||
>>no
>> ||
>> 
>> 
>> == Sponsors ==
>> Champion
>> 
>> * Chris Mattmann (mattmann at apache dot org)
>> 
>> Nominated Mentors
>> 
>> * Chris A. Mattmann (mattmann at apache dot org)
>> * Chris Douglas (cdouglas at apache dot org)
>> * Paul Ramirez (pramirez at apache dot org)
>> 
>> Sponsoring Entity
>> 
>> * Apache Incubator
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by Suresh Marru <sm...@apache.org>.
Hi Chris,

Great proposal, I am personally looking forward to see more of these federal government and international collaboration efforts adopt open community process. Impressive to see this project is pulling together initial developers from government and universities of US, South Africa, UK, Germany and India (I will hope all of them will be able to get their CLA's cleared). You are really scraping the black ice on bureaucratic highways and preventing at least some reinventing wheels with valuable tax payers money in multiple countries. 

Will this system evaluate and be inclusive of the community earth system modeling community?

Suresh

On Jan 26, 2013, at 2:48 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:

> Hi Everyone!
> 
> I bring to the Incubator a new proposal for the Apache Open Climate
> Workbench (incubating) project. I've added the proposal to the Incubator
> wiki here:
> 
> http://wiki.apache.org/incubator/ClimateProposal
> 
> 
> The project is a distributed, scalable approach for the rapid comparison
> of remote sensing data (e.g., from NASA) with that of climate model output
> generated by major US and international activities including the US
> National Climate Assessment, the Coordinated Regional Downscaling
> Experiment (CORDEX), the North American Regional Climate Change Assessment
> Program (NARCCAP) the Intergovernmental Panel on Climate Change (IPCC).
> Much of the software to be donated and evolved here at Apache as a
> community includes core dependencies to several Apache projects (OODT,
> Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
> software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.
> 
> We welcome your feedback over the next week or so for discussion and look
> forward to it!
> 
> Cheers,
> Chris Mattmann
> Proposed Mentor and Champion
> 
> ---------------proposal text
> 
> = Apache Open Climate Workbench, tool for scalable comparison of remote
> sensing observations to climate model outputs, regionally and globally. =
> === Abstract ===
> The Apache Open Climate Workbench proposal desires to contribute an
> existing community of software related to the analysis and evaluation of
> climate models, and related to the use of remote sensing data in that
> process. 
> 
> Specifically, we will bring a fundamental software toolkit for analysis
> and evaluation of climate model output against remote sensing data. The
> toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
> Evaluation System (RCMES)]]. RCMES provides two fundamental components for
> the easy, intuitive comparison of climate model output against remote
> sensing data. The first component called RCMED (for "Regional Climate
> Model Evaluation Database") is a scalable cloud database that decimates
> remote sensing data and renalysis data related to climate using Apache
> OODT extractors, Apache Tika, etc. These transformations make
> traditionally heterogeneous upstream remote sensing data and climate model
> output homogeneous and unify them into a data point model of the form
> (lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
> and Longitude (lng) are in WGS84 format, but can be reformatted on the
> fly. time is in ISO 8601 format, a string sortable format independent of
> underlying store. value carries with it units, related to interpretation
> and height allows for different values for different atmospheric vertical
> levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
> and Apache Hive, along with hooks to PostGIS and MySQL (traditional
> relational databases). The second component of the system, RCMET (for
> "Regional Climate Model Evaluation Toolkit") provides facilities for
> connecting to RCMED, dynamically obtaining remote sensing data for a
> space/time region of interest, grabbing associated model output (that the
> user brings, or from the Earth System Grid Federation) of the same form,
> and then regridding the remote sensing data to be on the model output
> grid, or the model output to be on the remote sensing data grid. The
> regridded data spatially is then temporally regridded using techniques
> including seasonal cycle compositing (e.g., all summer months, all
> Januaries, etc.), or by daily, monthly, etc. The uniform model output and
> remote sensing data are then analyzed using pluggable metrics, e.g.,
> Probability Distribution Functions (PDFs), Root Mean Squared Error (RMSE),
> Bias, and other (possibly user-defined) techniques, computing an analyzed
> comparison or evaluation. This evaluation is then visualized by plugging
> in to the NCAR NCL library for producing static plots (histograms, time
> series, etc.)
> 
> We also have performed a great deal of work in packaging RCMES to make the
> system easy to deploy. We have working Virtual Machines (VMWare VMX and
> Virtual Box OVA compatible formats) and we also have an installer built on
> Python Buildout (http://buildout.org/) called "Easy RCMET" for dynamically
> constructing the RCMET toolkit.
> 
> RCMES is currently supporting a number of recognized climate projects of
> (inter-)national significance. In particular, RCMES is supporting the
> [[http://www.globalchange.gov/what-we-do/assessment|U.S. National Climate
> Assessment (NCA) activities]] on behalf of NASA's contribution to the NCA;
> is working with the [[http://www.narccap.ucar.edu/|North American Regional
> Climate Change Assessment Program (NARCCAP)]]; and is also working with
> the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
> Regional Downscaling Experiment (CORDEX)]].
> 
> === Proposal ===
> We propose to transition the RCMES software community, which includes
> developers of the RCMET and RCMED software, along with users of RCMES in
> the CORDEX project across a variety of academic institutions, scientists
> helping to improve the RCMES metrics, and visualizations, and regridding
> algorithms, packagers making RCMES easier to install, and scientists
> helping to lead some of these international projects that are already
> using RCMES. 
> 
> We have been working on the RCMES project since 2009 funded initially by
> the American Recovery and Reinvestment Act (ARRA) project out at NASA, and
> then branching out into other sources of support and sustainability (NASA;
> NSF, etc. -- see the
> [[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
> the RCMES website for a full list of supporting U.S. and international
> partners). 
> 
> With the existing RCMES community at Apache, we will also work to
> encourage other climate software projects e.g., Open Climate GIS, elements
> of the Earth System Grid Federation, other NASA climate projects funded
> under the Computational Modeling, Algorithms and Cyberinfrastructure
> (CMAC) to contribute to the Open Climate Workbench here at Apache.
> 
> RCMED is a Big Data project that combines several underlying Apache
> software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related data
> management software. Its primary language is Java; RCMET, on the other
> hand, is a Python API, associated set of classes (framework), set of
> Python Bottle Web services, and a PHP "Wizard"-based User Interface that
> leverages Apache OODT Balance.
> 
> === Background ===
> Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
> his solid experience with Apache OODT and bringing it to the ASF. Chris
> worked for a year to get the support of the JPL community including
> approvals from the Software Release authority at JPL to release the
> software. 
> 
> The initial code drop will include the RCMES SVN repository from JPL
> including prior revisions. We anticipate also including a smaller package,
> CDX, which contains some useful facilities for regridding, and command
> line tools for manipulating large datasets, and working with OPeNDAP, etc.
> 
> After the code drop, we will work with our developers, users, documentors,
> and other members of the team to teach those unfamiliar with the Apache
> way how it works around here at Apache. 30% of the community from RCMES
> includes those intimately familiar with Apache including 5 ASF members --
> the other 70% include a range of scientific code developers, climate
> scientists that use RCMES, program officers that will help make
> documentation and slides for the code, and advocate for it in the
> community. Their experience with Apache ranges from using various ASF
> products, to contributing patches to them, to not using any ASF software
> at all.
> 
> With this diversity, we anticipate that while everything may not just work
> turnkey out of the box, this represents a unique opportunity to
> demonstrate Apache to the international community and to show the benefits
> of its community and social models. That said, we also have a lot of ASF
> experience to make sure everyone learns the Apache way.
> 
> === Rationale ===
> We are bringing RCMES to Apache for a few reasons. First, we feel that it
> will immediately enable our collaborators across a number of institutions
> both nationally and internationally have the opportunity to work on a
> common software base, and to improve it with contributions from their own
> sites. Currently these are difficult to negotiate now because of varied
> legal and contribution frameworks -- Apache allows us to simplify this to
> a unified one. Second, using the ASF's world-wide mirroring system, we
> will be able to deliver climate software broadly to the community as we
> release it, rather than sneaker netting the software around or
> establishing our own point release infrastructure.
> 
> Bringing this project to Apache also immediately thrusts the ASF into the
> thriving ecosystem of the Coordinated Regional Downscaling Experiment
> (CORDEX), the US National Cimate Assessment, the North American Regional
> Climate Change Assessment Program (the US contribution to CORDEX) and into
> relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
> assessment activities at a number of different institutions. We also seek
> to help lead and encourage du jour standard development rather than top
> down level dictating of standards for climate software and the ASF will
> provide us a means for that.
> 
> === Initial Goals ===
> The initial goals of the proposed project are:
> 
> * Stand up a sustaining Apache-based community around the JPL RCMES
> codebase.
> * Active relationships and possible cooperation with related projects and
> communities, including end user and scientific communities, CORDEX,
> NARCCAP, US NCA, IPCC, ESG, etc.
> * Active relationships and possible cooperation with existing Apache
> communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
> * Initial Apache release.
> * Leverage Apache Open Climate Workbench in climate activities at NASA,
> in the international community as mentioned above, and beyond.
> * Vetting all software licenses and making sure IP is clear (software
> grant from JPL forthcoming).
> 
> == Current Status ==
> === Meritocracy ===
> 30% of the proposed initial committers are familiar with the meritocracy
> principles of Apache. As stated above this includes 5 ASF members. Of the
> mentorship list, we have included Chris Douglas, a PMC member from Hadoop
> and ASF member to help guide the community. Chris M. and Chris D. have
> guided a number of projects through the Incubator over the years. The
> other mentor includes Paul Ramirez, who has experience with the Incubator
> -- he was a mentor for Apache Any23, and also was  one of the PPMC members
> and eventual mentor for Apache SIS. The 70% of proposed initial committers
> that aren't as familiar with Apache have a broad range of experience in
> other open source projects, and have a deep respect and affinity for the
> foundation and the work that gets done here. The more experience ASF
> mentors and project members will help to guide them.
> === Community ===
> There is an existing, established community of developers and users of
> this projet. This includes established communities including the
> Coordinated Regional Downscaling Experiment, the U.S. National Climate
> Assessment (NCA), the North American Regional Climate Change Assessment
> Program (NARCCAP), and more. The Coordinated Regional Climate Downscaling
> Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
> world wide effort of coordination of regional climate downscaling (RCD)
> experiments driven by the World Climate Research Program (WRCP,
> http://www.wcrp-climate.org/index.shtml). Recently, a large number of RCD
> projects have been carried out on a large parts of the world. To maximize
> the benefits of these research activities the WCRP designed a framework
> (Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets of
> RCD-based information for the recent historical past and 21st century
> projections, covering the majority of populated land regions on the
> globe".  CORDEX defined different control domains (up to 10,
> http://cordex.dmi.dk/joomla/) for almost all the populated regions of the
> world in a way to standardize the experiments and make them comparable. A
> key region focused on Africa was also designated as the top priority by
> WRCP. CORDEX also provides a a series of conventions and list of variables
> that have to be followed by any project that wants to contribute to the
> experiment. Each CORDEX region has a coordinator and regional and
> international periodic meetings are scheduled in a way to ensure the
> global well being. NARCCAP is the U.S. contribution to CORDEX. From the
> [[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
> National Climate Assessment]] site, work is "being conducted under the
> auspices of the Global Change Research Act of 1990. The GCRA requires a
> report to the President and the Congress every four years that integrates,
> evaluates, and interprets the findings of the U.S. Global Change Research
> Program (USGCRP); analyzes the effects of global change on the natural
> environment, agriculture, energy production and use, land and water
> resources, transportation, human health and welfare, human social systems,
> and biological diversity; and analyzes current trends in global change,
> both human-induced and natural, and projects major trends for the
> subsequent 25 to 100 years."
> 
> Apache Open Climate Workbench will support all of these communities above,
> with an eye towards being a general purpose climate evaluation toolkit for
> model output and remote sensing data.
> 
> === Core Developers ===
> The initial set of developers comes from various NASA centers (JPL, and
> Goddard Space Flight Center), NASA HQ, various  Universities participating
> in CORDEX (Cape Town, University of New South Wales), the Indian Institute
> of Tropical Meteorology, the Free Univ. Berlin), the University of
> California Los Angeles, and Howard University. As mentioned previously
> several of our developers are Apache veterans and understand how it works
> around here and for those that don't, they will have great mentorship.
> 
> === Alignment ===
> Our proposed effort aligns with the U.S. National Climate Assessment, the
> CORDEX effort, other efforts, including the Earth System Grid Federation,
> other climate software including the Open Climate GIS toolkit, other
> science portals for climate including the Climate Information Portal (CIP)
> at the University of Cape Town, and other related projects.
> 
> There are also a number of related Apache projects and dependencies, that
> will be mentioned in the Relationships with Other Apache products section.
> 
> == Known Risks ==
> === Orphaned products ===
> Our project has a history of funding support from JPL, NASA (Applications
> program/ARRA, NCA, AIST), NSF (ExArch project), international investment
> from collaborators, and from other funding sources. The funding sources
> are all target future deliverables and activities, so there is little
> chance this software and community will be orphaned.
> 
> === Inexperience with Open Source ===
> All the initial developers have worked on open source before -- 30% of the
> proposed initial community are experience with the ASF, and are PMC
> members and committers on ASF project including 5 ASF members. Our mentors
> are all ASF members, and we welcome any interest from additional Apache
> mentors in the effort. Those 70% of our project that aren't Apache
> committers, PMC members, or members will benefit from the leadership of
> the other 30% of the project.
> 
> === Homogenous Developers ===
> The initial developers come from a variety of backgrounds and with a
> variety of needs for the proposed framework. Everyone is used to
> communicating on mailing lists as the project spans timezones,
> international institutions and centers of excellence for climate science.
> 
> === Reliance on Salaried Developers ===
> All of the proposed initial developers are paid to work on this or related
> projects, but the proposed project is not the primary task for anyone.
> 
> === Relationships with Other Apache Products ===
> As mentioned above, RCMES and the Apache Open Climate Workbench already
> depend on Apache OODT for facade interfaces to underlying data warehouses
> for storing remote sensing data; and for metadata extraction and
> transformation. The software also uses Apache Tika for this (through a
> transitive dependency from OODT). In addition, we have hooks to Apache
> Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
> remote sensing data from MySQL and into HIVE.
> 
> === A Excessive Fascination with the Apache Brand ===
> All of us are familiar with Apache and have a respect for its brand and
> community. Chris Mattmann is a big proponent of Apache's sustainability
> factor -- and it's ability to grow software communities, in an
> institution, or funding source neutral manner. All of the community have
> an extreme respect for Apache, including those in our communities who
> aren't necessarily trained computer scientists, but are Scientists (big
> "S", e.g., land, physical, Earth/Climate scientists).
> 
> == Documentation ==
> The initial RCMES code base will come from the internal JPL Subversion
> repository. The [[Regional Climate Model Evaluation System (RCMES)
> project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
> has documentation on the existing software, including links to funding
> support, communities, and other projects. We will continue to maintain
> that site at JPL, which is part of the reason for rebranding the project
> here at Apache with a new name to not interfere with the existing RCMES
> one that has a following. In addition, we hope to evolve RCMES@JPL to have
> increasing levels of dependency on Apache Open Climate Workbench, so that
> we can incrementally transition with little impact to existing customers.
> 
> In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange (CDX)]]
> website also has documentation on the existing software.
> 
> == Initial Source ==
> The project will start with seed code donated by NASA JPL via Mattmann and
> the rest of the initial committers, which consists of the Regional Climate
> Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
> (CDX) software. This will include the core Python API for RCMET, the RCMED
> OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
> and HIVE), and the RCMED extractors for various climate formats. The
> source will also include Easy-RCMET, the Python Buildout for RCMET. In
> addition, we will bring along the CDX toolkit, which includes a CDX client
> package that performs subsetting, access, regridding of climate data; and
> also includes a Python Buildout installer of its own called Uber CDX.
> 
> == Source and Intellectual Property Submission Plan ==
> All seed code and other contributions will be handled through the normal
> Apache contribution process. Mattmann has been authorized by NASA JPL to
> lead the contribution of RCMES and CDX into the Incubator via his existing
> Apache CLA, and a Software Grant to be provided.
> 
> We will also contact other related efforts for possible cooperation and
> contributions.
> 
> == External Dependencies ==
> Our project depends on a number of external libraries with various
> licensing conditions. An initial list of such dependencies is shown below.
> ||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
> ||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
> ||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT compat||
> ||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
> ||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modified
> PSF license||
> ||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
> ||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
> ||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
> ||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||MI
> T||
> 
> 
> == Cryptography ==
> The project itself will not use cryptography, but it is possible that some
> of the external software libraries will include cryptographic code to
> handle features present in various science data formats. If we need to
> provide an export control statement regarding cryptographic code per
> Apache policy, we will follow a similar approach by Mattmann in
> [[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting lead
> this effort in Apache Tika. Mattmann is familiar with this process.
> 
> == Required Resources ==
> Mailing lists
> 
> * dev@climate.incubator.apache.org
> * commits@climate.incubator.apache.org
> * private@climate.incubator.apache.org
> 
> Subversion Directory
> 
> * https://svn.apache.org/repos/asf/incubator/climate
> 
> Issue Tracking
> 
> * JIRA CLIMATE (CLIMATE)
> 
> Other Resources
> 
> * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
> * Review Board instance - CLIMATE
> * Jenkins instance - CLIMATE
> 
> == Initial Committers ==
> ||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
> ||Chris A. Mattmann ||mattmann at apache dot org
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
> ||Cameron E. Goodale ||cgoodale at apache dot org
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
> ||Paul Ramirez ||pramirez at apache dog org
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
> ||Andrew F. Hart ||ahart at apache dot org
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
> ||Jinwon Kim||jkim at atmos dot ucla dot edu
> ||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
> System Science and Engineering]] || no||
> ||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
> ||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
> gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
> ||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
> ||Daniel J. Crichton||crichton at apache dot
> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
> ||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
> ||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.html
> |Howard University]] || no ||
> ||Paul Zimdars||pzimdars at apache dot
> org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
> ||Chris Jack||cjack at csag dot uct dot ac dot
> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
> ||Bruce Hewitson||hewitson at csag dot uct dot ac dot
> za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
> ||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
> au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
> ||Jason Evans||jason dot evans at unsw dot edu dot
> au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
> ||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
> ||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
> ||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
> ||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
> ||J. Sanjay||sanjay at tropmet dot res dot in ||
> [[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || yes
> ||
> ||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
> ||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
> yes ||
> ||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
> [[http://hq.nassa.gov/|NASA HQ]] || no ||
> ||Laura Carriere||laura dot carriere at nasa dot gov
> ||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no ||
> ||Denis Nadeau|| denis dot nadeau at nasa dot
> gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
> ||
> 
> 
> == Sponsors ==
> Champion
> 
> * Chris Mattmann (mattmann at apache dot org)
> 
> Nominated Mentors
> 
> * Chris A. Mattmann (mattmann at apache dot org)
> * Chris Douglas (cdouglas at apache dot org)
> * Paul Ramirez (pramirez at apache dot org)
> 
> Sponsoring Entity
> 
> * Apache Incubator
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Dear Nick,

On 1/27/13 10:34 PM, "Nick Kew" <ni...@apache.org> wrote:

>
>On 26 Jan 2013, at 19:48, Mattmann, Chris A (388J) wrote:
>
>> Hi Everyone!
>> 
>> I bring to the Incubator a new proposal for the Apache Open Climate
>> Workbench (incubating) project. I've added the proposal to the Incubator
>> wiki here:
>> 
>> http://wiki.apache.org/incubator/ClimateProposal
>
>Sounds quite exciting!  I'd like to come on board, though whether as an
>active participant or passively will depend on time and circumstance.

You are welcome to join and we'd be happy to have you as a mentor, a PPMC
member, or a
committer. Your wisdom will be great.

>
>Some years back I spent a lot of time&effort on the somewhat-related
>task of developing systems, firstly to produce, archive and distribute
>some of the datasets (mostly the Global AVHRR 1Km series in the '90s)
>and secondly to develop tools to visualise different datasets alongside
>and superimposed on each other.

Awesome, so you were dealing with remote sensing stuff there, cool.

>
>I see your proposal has a slightly different focus (and obviously benefits
>from a new generation of infrastructure in which a terabyte is no longer
>a mindblowing amount of data to manage), but I hope my background
>might still serve to kick-start an involvement!

Yep that would be awesome welcome aboard!

Cheers,
Chris

>
>-- 
>Nick Kew
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Dear Nick,

Please add yourself to the proposal on the wiki as a mentor and/or PPMC
member and committer in the proposal table. I will call a VOTE next week
on Monday so if you can do it before then, thanks. Otherwise I'll do it
but I don't know your affiliation.

Cheers,
Chris

On 1/27/13 10:34 PM, "Nick Kew" <ni...@apache.org> wrote:

>
>On 26 Jan 2013, at 19:48, Mattmann, Chris A (388J) wrote:
>
>> Hi Everyone!
>> 
>> I bring to the Incubator a new proposal for the Apache Open Climate
>> Workbench (incubating) project. I've added the proposal to the Incubator
>> wiki here:
>> 
>> http://wiki.apache.org/incubator/ClimateProposal
>
>Sounds quite exciting!  I'd like to come on board, though whether as an
>active participant or passively will depend on time and circumstance.
>
>Some years back I spent a lot of time&effort on the somewhat-related
>task of developing systems, firstly to produce, archive and distribute
>some of the datasets (mostly the Global AVHRR 1Km series in the '90s)
>and secondly to develop tools to visualise different datasets alongside
>and superimposed on each other.
>
>I see your proposal has a slightly different focus (and obviously benefits
>from a new generation of infrastructure in which a terabyte is no longer
>a mindblowing amount of data to manage), but I hope my background
>might still serve to kick-start an involvement!
>
>-- 
>Nick Kew
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by Nick Kew <ni...@apache.org>.
On 26 Jan 2013, at 19:48, Mattmann, Chris A (388J) wrote:

> Hi Everyone!
> 
> I bring to the Incubator a new proposal for the Apache Open Climate
> Workbench (incubating) project. I've added the proposal to the Incubator
> wiki here:
> 
> http://wiki.apache.org/incubator/ClimateProposal

Sounds quite exciting!  I'd like to come on board, though whether as an
active participant or passively will depend on time and circumstance.

Some years back I spent a lot of time&effort on the somewhat-related
task of developing systems, firstly to produce, archive and distribute
some of the datasets (mostly the Global AVHRR 1Km series in the '90s)
and secondly to develop tools to visualise different datasets alongside
and superimposed on each other.

I see your proposal has a slightly different focus (and obviously benefits
from a new generation of infrastructure in which a terabyte is no longer
a mindblowing amount of data to manage), but I hope my background
might still serve to kick-start an involvement!

-- 
Nick Kew


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Re: [DISCUSS] Apache Open Climate Workbench

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi All,

Thanks for the community feedback so far.

Since conversation is stabilizing, I am going to call for an official VOTE
starting on Monday early AM PT. I'll let it run through the whole week,
closing it around Friday EOD PT, and then we can go from there depending
on the outcome.

Thank you and have a great weekend.

Cheers,
Chris

On 1/26/13 11:48 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:

>Hi Everyone!
>
>I bring to the Incubator a new proposal for the Apache Open Climate
>Workbench (incubating) project. I've added the proposal to the Incubator
>wiki here:
>
>http://wiki.apache.org/incubator/ClimateProposal
>
>
>The project is a distributed, scalable approach for the rapid comparison
>of remote sensing data (e.g., from NASA) with that of climate model output
>generated by major US and international activities including the US
>National Climate Assessment, the Coordinated Regional Downscaling
>Experiment (CORDEX), the North American Regional Climate Change Assessment
>Program (NARCCAP) the Intergovernmental Panel on Climate Change (IPCC).
>Much of the software to be donated and evolved here at Apache as a
>community includes core dependencies to several Apache projects (OODT,
>Hadoop/HIVE, Sqoop, Tika), and to several key toolkits in the Python
>software community including NumPy, SciPy, Matplotlib, NCAR NCL, etc.
>
>We welcome your feedback over the next week or so for discussion and look
>forward to it!
>
>Cheers,
>Chris Mattmann
>Proposed Mentor and Champion
>
>---------------proposal text
>
>= Apache Open Climate Workbench, tool for scalable comparison of remote
>sensing observations to climate model outputs, regionally and globally. =
>=== Abstract ===
>The Apache Open Climate Workbench proposal desires to contribute an
>existing community of software related to the analysis and evaluation of
>climate models, and related to the use of remote sensing data in that
>process. 
>
>Specifically, we will bring a fundamental software toolkit for analysis
>and evaluation of climate model output against remote sensing data. The
>toolkit is called the [[http://rcmes.jpl.nasa.gov|Regional Climate Model
>Evaluation System (RCMES)]]. RCMES provides two fundamental components for
>the easy, intuitive comparison of climate model output against remote
>sensing data. The first component called RCMED (for "Regional Climate
>Model Evaluation Database") is a scalable cloud database that decimates
>remote sensing data and renalysis data related to climate using Apache
>OODT extractors, Apache Tika, etc. These transformations make
>traditionally heterogeneous upstream remote sensing data and climate model
>output homogeneous and unify them into a data point model of the form
>(lat, lng, time, value, height) on a per parameter basis. Latitude (lat)
>and Longitude (lng) are in WGS84 format, but can be reformatted on the
>fly. time is in ISO 8601 format, a string sortable format independent of
>underlying store. value carries with it units, related to interpretation
>and height allows for different values for different atmospheric vertical
>levels. All of RCMES is built on Apache OODT, Apache Sqoop/Apache Hadoop
>and Apache Hive, along with hooks to PostGIS and MySQL (traditional
>relational databases). The second component of the system, RCMET (for
>"Regional Climate Model Evaluation Toolkit") provides facilities for
>connecting to RCMED, dynamically obtaining remote sensing data for a
>space/time region of interest, grabbing associated model output (that the
>user brings, or from the Earth System Grid Federation) of the same form,
>and then regridding the remote sensing data to be on the model output
>grid, or the model output to be on the remote sensing data grid. The
>regridded data spatially is then temporally regridded using techniques
>including seasonal cycle compositing (e.g., all summer months, all
>Januaries, etc.), or by daily, monthly, etc. The uniform model output and
>remote sensing data are then analyzed using pluggable metrics, e.g.,
>Probability Distribution Functions (PDFs), Root Mean Squared Error (RMSE),
>Bias, and other (possibly user-defined) techniques, computing an analyzed
>comparison or evaluation. This evaluation is then visualized by plugging
>in to the NCAR NCL library for producing static plots (histograms, time
>series, etc.)
>
>We also have performed a great deal of work in packaging RCMES to make the
>system easy to deploy. We have working Virtual Machines (VMWare VMX and
>Virtual Box OVA compatible formats) and we also have an installer built on
>Python Buildout (http://buildout.org/) called "Easy RCMET" for dynamically
>constructing the RCMET toolkit.
>
>RCMES is currently supporting a number of recognized climate projects of
>(inter-)national significance. In particular, RCMES is supporting the
>[[http://www.globalchange.gov/what-we-do/assessment|U.S. National Climate
>Assessment (NCA) activities]] on behalf of NASA's contribution to the NCA;
>is working with the [[http://www.narccap.ucar.edu/|North American Regional
>Climate Change Assessment Program (NARCCAP)]]; and is also working with
>the International [[http://wcrp-cordex.ipsl.jussieu.fr/|Coordinated
>Regional Downscaling Experiment (CORDEX)]].
>
>=== Proposal ===
>We propose to transition the RCMES software community, which includes
>developers of the RCMET and RCMED software, along with users of RCMES in
>the CORDEX project across a variety of academic institutions, scientists
>helping to improve the RCMES metrics, and visualizations, and regridding
>algorithms, packagers making RCMES easier to install, and scientists
>helping to lead some of these international projects that are already
>using RCMES. 
>
>We have been working on the RCMES project since 2009 funded initially by
>the American Recovery and Reinvestment Act (ARRA) project out at NASA, and
>then branching out into other sources of support and sustainability (NASA;
>NSF, etc. -- see the
>[[http://rcmes.jpl.nasa.gov/about/overview|acknowledgements]] section on
>the RCMES website for a full list of supporting U.S. and international
>partners). 
>
>With the existing RCMES community at Apache, we will also work to
>encourage other climate software projects e.g., Open Climate GIS, elements
>of the Earth System Grid Federation, other NASA climate projects funded
>under the Computational Modeling, Algorithms and Cyberinfrastructure
>(CMAC) to contribute to the Open Climate Workbench here at Apache.
>
>RCMED is a Big Data project that combines several underlying Apache
>software -- OODT, Tika, Hadoop, HIVE, and Sqoop -- and other related data
>management software. Its primary language is Java; RCMET, on the other
>hand, is a Python API, associated set of classes (framework), set of
>Python Bottle Web services, and a PHP "Wizard"-based User Interface that
>leverages Apache OODT Balance.
>
>=== Background ===
>Bringing RCMES to Apache was the brain-child of Chris Mattmann, based on
>his solid experience with Apache OODT and bringing it to the ASF. Chris
>worked for a year to get the support of the JPL community including
>approvals from the Software Release authority at JPL to release the
>software. 
>
>The initial code drop will include the RCMES SVN repository from JPL
>including prior revisions. We anticipate also including a smaller package,
>CDX, which contains some useful facilities for regridding, and command
>line tools for manipulating large datasets, and working with OPeNDAP, etc.
>
>After the code drop, we will work with our developers, users, documentors,
>and other members of the team to teach those unfamiliar with the Apache
>way how it works around here at Apache. 30% of the community from RCMES
>includes those intimately familiar with Apache including 5 ASF members --
>the other 70% include a range of scientific code developers, climate
>scientists that use RCMES, program officers that will help make
>documentation and slides for the code, and advocate for it in the
>community. Their experience with Apache ranges from using various ASF
>products, to contributing patches to them, to not using any ASF software
>at all.
>
>With this diversity, we anticipate that while everything may not just work
>turnkey out of the box, this represents a unique opportunity to
>demonstrate Apache to the international community and to show the benefits
>of its community and social models. That said, we also have a lot of ASF
>experience to make sure everyone learns the Apache way.
>
>=== Rationale ===
>We are bringing RCMES to Apache for a few reasons. First, we feel that it
>will immediately enable our collaborators across a number of institutions
>both nationally and internationally have the opportunity to work on a
>common software base, and to improve it with contributions from their own
>sites. Currently these are difficult to negotiate now because of varied
>legal and contribution frameworks -- Apache allows us to simplify this to
>a unified one. Second, using the ASF's world-wide mirroring system, we
>will be able to deliver climate software broadly to the community as we
>release it, rather than sneaker netting the software around or
>establishing our own point release infrastructure.
>
>Bringing this project to Apache also immediately thrusts the ASF into the
>thriving ecosystem of the Coordinated Regional Downscaling Experiment
>(CORDEX), the US National Cimate Assessment, the North American Regional
>Climate Change Assessment Program (the US contribution to CORDEX) and into
>relevance for upcoming Intergovernmental Panel on Climate Change (IPCC)
>assessment activities at a number of different institutions. We also seek
>to help lead and encourage du jour standard development rather than top
>down level dictating of standards for climate software and the ASF will
>provide us a means for that.
>
>=== Initial Goals ===
>The initial goals of the proposed project are:
>
> * Stand up a sustaining Apache-based community around the JPL RCMES
>codebase.
> * Active relationships and possible cooperation with related projects and
>communities, including end user and scientific communities, CORDEX,
>NARCCAP, US NCA, IPCC, ESG, etc.
> * Active relationships and possible cooperation with existing Apache
>communities, e.g., OODT, Hadoop/HIVE, Sqoop, Tika, SIS, etc.
> * Initial Apache release.
> * Leverage Apache Open Climate Workbench in climate activities at NASA,
>in the international community as mentioned above, and beyond.
> * Vetting all software licenses and making sure IP is clear (software
>grant from JPL forthcoming).
> 
>== Current Status ==
>=== Meritocracy ===
>30% of the proposed initial committers are familiar with the meritocracy
>principles of Apache. As stated above this includes 5 ASF members. Of the
>mentorship list, we have included Chris Douglas, a PMC member from Hadoop
>and ASF member to help guide the community. Chris M. and Chris D. have
>guided a number of projects through the Incubator over the years. The
>other mentor includes Paul Ramirez, who has experience with the Incubator
>-- he was a mentor for Apache Any23, and also was  one of the PPMC members
>and eventual mentor for Apache SIS. The 70% of proposed initial committers
>that aren't as familiar with Apache have a broad range of experience in
>other open source projects, and have a deep respect and affinity for the
>foundation and the work that gets done here. The more experience ASF
>mentors and project members will help to guide them.
>=== Community ===
>There is an existing, established community of developers and users of
>this projet. This includes established communities including the
>Coordinated Regional Downscaling Experiment, the U.S. National Climate
>Assessment (NCA), the North American Regional Climate Change Assessment
>Program (NARCCAP), and more. The Coordinated Regional Climate Downscaling
>Experiment (CORDEX, http://wcrp.ipsl.jussieu.fr/cordex/about.html) is a
>world wide effort of coordination of regional climate downscaling (RCD)
>experiments driven by the World Climate Research Program (WRCP,
>http://www.wcrp-climate.org/index.shtml). Recently, a large number of RCD
>projects have been carried out on a large parts of the world. To maximize
>the benefits of these research activities the WCRP designed a framework
>(Giorgi, WMO-Bulletin, 2009) focused on "quality-control [of] data sets of
>RCD-based information for the recent historical past and 21st century
>projections, covering the majority of populated land regions on the
>globe".  CORDEX defined different control domains (up to 10,
>http://cordex.dmi.dk/joomla/) for almost all the populated regions of the
>world in a way to standardize the experiments and make them comparable. A
>key region focused on Africa was also designated as the top priority by
>WRCP. CORDEX also provides a a series of conventions and list of variables
>that have to be followed by any project that wants to contribute to the
>experiment. Each CORDEX region has a coordinator and regional and
>international periodic meetings are scheduled in a way to ensure the
>global well being. NARCCAP is the U.S. contribution to CORDEX. From the
>[[http://www.globalchange.gov/what-we-do/assessment/nca-overview|US
>National Climate Assessment]] site, work is "being conducted under the
>auspices of the Global Change Research Act of 1990. The GCRA requires a
>report to the President and the Congress every four years that integrates,
>evaluates, and interprets the findings of the U.S. Global Change Research
>Program (USGCRP); analyzes the effects of global change on the natural
>environment, agriculture, energy production and use, land and water
>resources, transportation, human health and welfare, human social systems,
>and biological diversity; and analyzes current trends in global change,
>both human-induced and natural, and projects major trends for the
>subsequent 25 to 100 years."
>
>Apache Open Climate Workbench will support all of these communities above,
>with an eye towards being a general purpose climate evaluation toolkit for
>model output and remote sensing data.
>
>=== Core Developers ===
>The initial set of developers comes from various NASA centers (JPL, and
>Goddard Space Flight Center), NASA HQ, various  Universities participating
>in CORDEX (Cape Town, University of New South Wales), the Indian Institute
>of Tropical Meteorology, the Free Univ. Berlin), the University of
>California Los Angeles, and Howard University. As mentioned previously
>several of our developers are Apache veterans and understand how it works
>around here and for those that don't, they will have great mentorship.
>
>=== Alignment ===
>Our proposed effort aligns with the U.S. National Climate Assessment, the
>CORDEX effort, other efforts, including the Earth System Grid Federation,
>other climate software including the Open Climate GIS toolkit, other
>science portals for climate including the Climate Information Portal (CIP)
>at the University of Cape Town, and other related projects.
>
>There are also a number of related Apache projects and dependencies, that
>will be mentioned in the Relationships with Other Apache products section.
>
>== Known Risks ==
>=== Orphaned products ===
>Our project has a history of funding support from JPL, NASA (Applications
>program/ARRA, NCA, AIST), NSF (ExArch project), international investment
>from collaborators, and from other funding sources. The funding sources
>are all target future deliverables and activities, so there is little
>chance this software and community will be orphaned.
>
>=== Inexperience with Open Source ===
>All the initial developers have worked on open source before -- 30% of the
>proposed initial community are experience with the ASF, and are PMC
>members and committers on ASF project including 5 ASF members. Our mentors
>are all ASF members, and we welcome any interest from additional Apache
>mentors in the effort. Those 70% of our project that aren't Apache
>committers, PMC members, or members will benefit from the leadership of
>the other 30% of the project.
>
>=== Homogenous Developers ===
>The initial developers come from a variety of backgrounds and with a
>variety of needs for the proposed framework. Everyone is used to
>communicating on mailing lists as the project spans timezones,
>international institutions and centers of excellence for climate science.
>
>=== Reliance on Salaried Developers ===
>All of the proposed initial developers are paid to work on this or related
>projects, but the proposed project is not the primary task for anyone.
>
>=== Relationships with Other Apache Products ===
>As mentioned above, RCMES and the Apache Open Climate Workbench already
>depend on Apache OODT for facade interfaces to underlying data warehouses
>for storing remote sensing data; and for metadata extraction and
>transformation. The software also uses Apache Tika for this (through a
>transitive dependency from OODT). In addition, we have hooks to Apache
>Hadoop/HIVE, as well as dependencies on Apache Sqoop for dumping out
>remote sensing data from MySQL and into HIVE.
>
>=== A Excessive Fascination with the Apache Brand ===
>All of us are familiar with Apache and have a respect for its brand and
>community. Chris Mattmann is a big proponent of Apache's sustainability
>factor -- and it's ability to grow software communities, in an
>institution, or funding source neutral manner. All of the community have
>an extreme respect for Apache, including those in our communities who
>aren't necessarily trained computer scientists, but are Scientists (big
>"S", e.g., land, physical, Earth/Climate scientists).
>
>== Documentation ==
>The initial RCMES code base will come from the internal JPL Subversion
>repository. The [[Regional Climate Model Evaluation System (RCMES)
>project|http://rcmes.jpl.nasa.gov]] at [[JPL|http://www.jpl.nasa.gov/]]
>has documentation on the existing software, including links to funding
>support, communities, and other projects. We will continue to maintain
>that site at JPL, which is part of the reason for rebranding the project
>here at Apache with a new name to not interfere with the existing RCMES
>one that has a following. In addition, we hope to evolve RCMES@JPL to have
>increasing levels of dependency on Apache Open Climate Workbench, so that
>we can incrementally transition with little impact to existing customers.
>
>In addition, JPL's [[http://cdx.jpl.nasa.gov|Climate Data eXchange (CDX)]]
>website also has documentation on the existing software.
>
>== Initial Source ==
>The project will start with seed code donated by NASA JPL via Mattmann and
>the rest of the initial committers, which consists of the Regional Climate
>Model Evaluation System (RCMES) toolkit, and the Climate Data eXchange
>(CDX) software. This will include the core Python API for RCMET, the RCMED
>OODT catalog project (which stores remote sensing data to MySQL/PostGIS,
>and HIVE), and the RCMED extractors for various climate formats. The
>source will also include Easy-RCMET, the Python Buildout for RCMET. In
>addition, we will bring along the CDX toolkit, which includes a CDX client
>package that performs subsetting, access, regridding of climate data; and
>also includes a Python Buildout installer of its own called Uber CDX.
>
>== Source and Intellectual Property Submission Plan ==
>All seed code and other contributions will be handled through the normal
>Apache contribution process. Mattmann has been authorized by NASA JPL to
>lead the contribution of RCMES and CDX into the Incubator via his existing
>Apache CLA, and a Software Grant to be provided.
>
>We will also contact other related efforts for possible cooperation and
>contributions.
>
>== External Dependencies ==
>Our project depends on a number of external libraries with various
>licensing conditions. An initial list of such dependencies is shown below.
>||<tableclass="bodyTable"rowclass="b">'''Library''' ||'''License''' ||
>||<rowclass="b">[[http://www.ncl.ucar.edu/|NCAR NCL]]||MIT compat||
>||<rowclass="a">[[http://www.pyngl.ucar.edu/Nio.shtml|PyNIO]]||MIT
>compat||
>||<rowclass="b">[[http://www.pyngl.ucar.edu/|PyNGL]]||MIT compat||
>||<rowclass="a">[[http://matplotlib.sourceforge.net/|Matplotlib]]||Modifie
>d
> PSF license||
>||<rowclass="b">[[http://scipy.org/|Scipy]]||MIT compat||
>||<rowclass="a">[[http://numpy.scipy.org/|NumPy]]||MIT compat||
>||<rowclass="b">[[http://www.hdfgroup.org/HDF5/|HDF5]]||BSD||
>||<rowclass="a">[[http://www.unidata.ucar.edu/software/netcdf/|NetCDF]]||M
>I
>T||
>
>
>== Cryptography ==
>The project itself will not use cryptography, but it is possible that some
>of the external software libraries will include cryptographic code to
>handle features present in various science data formats. If we need to
>provide an export control statement regarding cryptographic code per
>Apache policy, we will follow a similar approach by Mattmann in
>[[http://lucene.apache.org/nutch/|Apache Nutch]] and by Jukka Zitting lead
>this effort in Apache Tika. Mattmann is familiar with this process.
>
>== Required Resources ==
>Mailing lists
>
> * dev@climate.incubator.apache.org
> * commits@climate.incubator.apache.org
> * private@climate.incubator.apache.org
>
>Subversion Directory
>
> * https://svn.apache.org/repos/asf/incubator/climate
>
>Issue Tracking
>
> * JIRA CLIMATE (CLIMATE)
>
>Other Resources
>
> * CLIMATE Wiki http://cwiki.apache.org/CLIMATE
> * Review Board instance - CLIMATE
> * Jenkins instance - CLIMATE
>
>== Initial Committers ==
>||'''Name''' ||'''Email''' ||'''Affiliation''' ||'''CLA''' ||
>||Chris A. Mattmann ||mattmann at apache dot org
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>||Cameron E. Goodale ||cgoodale at apache dot org
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>||Paul Ramirez ||pramirez at apache dog org
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>||Andrew F. Hart ||ahart at apache dot org
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] ||yes ||
>||Jinwon Kim||jkim at atmos dot ucla dot edu
>||[[http://jifresse.ucla.edu|UCLA Joint Institute for Regional Earth
>System Science and Engineering]] || no||
>||Duane Waliser||duane dot waliser at jpl dot nasa dot gov
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>||Huikyo Lee||Huikyo dot Lee at jpl dot nasa dot
>gov||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>||Paul Loikith|| Paul dot C dot Loikith at jpl dot nasa dot gov
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || no ||
>||Daniel J. Crichton||crichton at apache dot
>org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
>||Kim Whitehall||Kim dot D dot Whitehall at jpl dot nasa dot gov
>||[[http://www.physics1.howard.edu/~pmisra/HUPAS/HUPAS/HUPAS%20Jenkins.htm
>l
>|Howard University]] || no ||
>||Paul Zimdars||pzimdars at apache dot
>org||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
>||Chris Jack||cjack at csag dot uct dot ac dot
>za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>||Bruce Hewitson||hewitson at csag dot uct dot ac dot
>za||[[http://www.csag.uct.ac.za/|University of Cape Town]] || no ||
>||Lluis Fita Borrell||l dot fitaborrell at unsw dot edu dot
>au||[[http://unsw.edu.au/|University of New South Wales]] || yes ||
>||Jason Evans||jason dot evans at unsw dot edu dot
>au||[[http://unsw.edu.au/|University of New South Wales]] || no ||
>||Estani Gonzalez||estanislao dot gonzalez at met dot fu-berlin dot de
>||[[http://www.geo.fu-berlin.de/met/|Free University Berlin]] || yes ||
>||Luca Cinquini||luca dot cinquini at jpl dot nasa dot gov
>||[[http://www.jpl.nasa.gov/|NASA Jet Propulsion Laboratory]] || yes ||
>||J. Sanjay||sanjay at tropmet dot res dot in ||
>[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] || yes
>||
>||M. V. S. Rama Rao||ramarao at tropmet dot res dot in
>||[[http://tropmet.res.in/|Indian Institute of Tropical Meteorology]] ||
>yes ||
>||Tsengdar Lee||tsengdar dot j dot lee at nasa dot gov ||
>[[http://hq.nassa.gov/|NASA HQ]] || no ||
>||Laura Carriere||laura dot carriere at nasa dot gov
>||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no ||
>||Denis Nadeau|| denis dot nadeau at nasa dot
>gov||[[http://www.gsfc.nasa.gov/|NASA Goddard Space Flight Center]] || no
>||
>
>
>== Sponsors ==
>Champion
>
> * Chris Mattmann (mattmann at apache dot org)
>
>Nominated Mentors
>
> * Chris A. Mattmann (mattmann at apache dot org)
> * Chris Douglas (cdouglas at apache dot org)
> * Paul Ramirez (pramirez at apache dot org)
>
>Sponsoring Entity
>
> * Apache Incubator
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org