You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov> on 2013/06/20 16:37:42 UTC
[DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Hey Travis,
I would strongly urge you to do development on Apache SIS on Apache
hardware.
Github is great; and convenience. But when you commit there, we don't get
email notifications and so forth here and the community loses out (and we
lose
out) on having email records; archives, and other things here that show
work
is going on in SIS.
I have a simple proposal :) You guys are definitely more Git fans now than
SVN fans. Martin D when he originally came onto the project wanted to use
Git, and was more familiar with it, but took great effort to adopt SVN b/c
ASF support for Git at that time was quite limited.
However, with you here now; with Adam; with Martin; and with a number of
other folks contributing (Joe W. are you a Git guy?) that are Git fans,
it's worth revisiting this discussion. However, *after* 0.3 :) Let's
release
that using SVN so we don't hold that off anymore. After 0.3 maybe we can
move to Git if this discussion is favorable. Apache now supports writeable
Git repos (see http://git.apache.org/) and the project's canonical
repository
can be Git. We can still mirror to Github, etc., but the bits (and really
the
work) ought to be happening here at the ASF.
So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Thursday, June 20, 2013 7:31 AM
To: dev <de...@sis.apache.org>
Subject: Re: shapefile branch
>Good to know about the OGC/ISO interfaces.
>
>It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
>files
>etc. I can set up in another code repo on github. The reason I want to
>work
>on that concurrently is to stress test the existing library with lots of
>data to find bugs that may not appear with simple unit tests.
>
>
>
>Thanks,
>Travis
>
>
>
>
>
>On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
>martin.desruisseaux@geomatys.fr> wrote:
>
>> Le 20/06/13 12:47, Travis L Pinney a écrit :
>>
>> The java.util.Map is fairly basic now. An improvement could be a
>>feature
>>> class that has a map of <String, DataType>, where DataType corresponds
>>>to
>>> the appropriate DataType (
>>>
>>>http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html<h
>>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
>>> .)
>>> Currently I am converting everything to strings.
>>>
>>
>> Actually Feature, FeatureType and related interfaces derived from
>>OGC/ISO
>> standards (in particular GML - Geographic Markup Language - schemas) are
>> already provided in GeoAPI:
>>
>> http://www.geoapi.org/**snapshot/pending/org/opengis/**
>>
>>feature/package-summary.html<http://www.geoapi.org/snapshot/pending/org/o
>>pengis/feature/package-summary.html>
>>
>> This is in the "pending" part of GeoAPI, so we have room for revising
>> them, in particular make sure that they are still in agreement with
>>latest
>> OGC/ISO standards. Then we would need to provide an implementation in
>>SIS,
>> porting Geotk classes when possible or appropriate. However there is a
>> somewhat long road before we reach that point, so it seems to me that
>>your
>> current approach (String in java.util.Map) is good in the main time.
>>
>>
>>
>> The bulk ingests would be an api where you can call a jar file from
>>> hadoop,
>>> give it appropriate directory to pull shapefiles in HDFS, and it would
>>> process each shapefile per mapper. The first ingest I am working on is
>>>a
>>> transformation of points to a 2D-histogram to get an idea of density of
>>> features of all the shapefiles. This could be extended to have
>>>different
>>> types of outputs (store in a database or more efficient format on hdfs)
>>>
>>
>> I would suggest to separate the two tasks. I think that the above is
>>what
>> we call a "processing", which is the subject of (yet an other) OGC
>> standard. Processing and DataStore should be independent, i.e. someone
>>may
>> want to apply the above processing on NetCDF files too... Maybe we can
>> focus on ShapefileStore first, and revisit processing later? Processings
>> will need DataStores first in order to perform their work anyway...
>>
>> Martin
>>
>>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Suresh Marru <sm...@apache.org>.
+ 1.
I still like svn for ASF projects but not opposed to git.
Suresh
On Jun 20, 2013, at 10:37 AM, "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov> wrote:
> Hey Travis,
>
> I would strongly urge you to do development on Apache SIS on Apache
> hardware.
> Github is great; and convenience. But when you commit there, we don't get
> email notifications and so forth here and the community loses out (and we
> lose
> out) on having email records; archives, and other things here that show
> work
> is going on in SIS.
>
> I have a simple proposal :) You guys are definitely more Git fans now than
> SVN fans. Martin D when he originally came onto the project wanted to use
> Git, and was more familiar with it, but took great effort to adopt SVN b/c
> ASF support for Git at that time was quite limited.
>
> However, with you here now; with Adam; with Martin; and with a number of
> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> release
> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> move to Git if this discussion is favorable. Apache now supports writeable
> Git repos (see http://git.apache.org/) and the project's canonical
> repository
> can be Git. We can still mirror to Github, etc., but the bits (and really
> the
> work) ought to be happening here at the ASF.
>
> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Travis L Pinney <tr...@gmail.com>
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Thursday, June 20, 2013 7:31 AM
> To: dev <de...@sis.apache.org>
> Subject: Re: shapefile branch
>
>> Good to know about the OGC/ISO interfaces.
>>
>> It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
>> files
>> etc. I can set up in another code repo on github. The reason I want to
>> work
>> on that concurrently is to stress test the existing library with lots of
>> data to find bugs that may not appear with simple unit tests.
>>
>>
>>
>> Thanks,
>> Travis
>>
>>
>>
>>
>>
>> On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
>> martin.desruisseaux@geomatys.fr> wrote:
>>
>>> Le 20/06/13 12:47, Travis L Pinney a écrit :
>>>
>>> The java.util.Map is fairly basic now. An improvement could be a
>>> feature
>>>> class that has a map of <String, DataType>, where DataType corresponds
>>>> to
>>>> the appropriate DataType (
>>>>
>>>> http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html<h
>>>> ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
>>>> .)
>>>> Currently I am converting everything to strings.
>>>>
>>>
>>> Actually Feature, FeatureType and related interfaces derived from
>>> OGC/ISO
>>> standards (in particular GML - Geographic Markup Language - schemas) are
>>> already provided in GeoAPI:
>>>
>>> http://www.geoapi.org/**snapshot/pending/org/opengis/**
>>>
>>> feature/package-summary.html<http://www.geoapi.org/snapshot/pending/org/o
>>> pengis/feature/package-summary.html>
>>>
>>> This is in the "pending" part of GeoAPI, so we have room for revising
>>> them, in particular make sure that they are still in agreement with
>>> latest
>>> OGC/ISO standards. Then we would need to provide an implementation in
>>> SIS,
>>> porting Geotk classes when possible or appropriate. However there is a
>>> somewhat long road before we reach that point, so it seems to me that
>>> your
>>> current approach (String in java.util.Map) is good in the main time.
>>>
>>>
>>>
>>> The bulk ingests would be an api where you can call a jar file from
>>>> hadoop,
>>>> give it appropriate directory to pull shapefiles in HDFS, and it would
>>>> process each shapefile per mapper. The first ingest I am working on is
>>>> a
>>>> transformation of points to a 2D-histogram to get an idea of density of
>>>> features of all the shapefiles. This could be extended to have
>>>> different
>>>> types of outputs (store in a database or more efficient format on hdfs)
>>>>
>>>
>>> I would suggest to separate the two tasks. I think that the above is
>>> what
>>> we call a "processing", which is the subject of (yet an other) OGC
>>> standard. Processing and DataStore should be independent, i.e. someone
>>> may
>>> want to apply the above processing on NetCDF files too... Maybe we can
>>> focus on ShapefileStore first, and revisit processing later? Processings
>>> will need DataStores first in order to perform their work anyway...
>>>
>>> Martin
>>>
>>>
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
+1, makes sense Martin.
SVN should stay for now, and then maybe later Git if it makes sense.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Thursday, June 20, 2013 1:44 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
>I think that a migration to git would be desirable in medium term. But
>on my side, I need to learn git more, in particular how to manage
>branches with git. The DVS that I know was Mercurial actually.
>
>SVN works reasonably well for now. However merging is not yet as
>efficient than Mercurial (and presumably Git). In particular, giving the
>same file copied on 2 branches:
>
> branches/A/MyFile
> branches/B/MyFile
>
>If we rename MyFile on branch A, then merge with branch B, the changes
>done in that file on branch B are lost with SVN, while they are
>preserved with Mercurial. Until now I took that limitation as an
>incitative to think harder about the module/package/class names in the
>first place, and to not move lightly.
>
>Before I could switch to git, I would need to learn how to perform the
>work equivalent to Mercurial's "named branches". Git does not exactly
>have the Mercurial concept of named branches, but have something else
>providing similar functionality (I think). We also need to investigate
>about what will happen to directories other than the standard "trunk",
>"branches" and "tags". We have "data", "ip-review", "presentations" and
>"site". I'm not sure if those directories are already on the Git clone.
>
>In summary: migration to Git would requires some work (I think), so
>maybe it could be a medium term goal, somewhere after the 0.3 release?
>
> Martin
>
>
>Le 20/06/13 16:37, Mattmann, Chris A (398J) a écrit :
>> I have a simple proposal :) You guys are definitely more Git fans now
>>than
>> SVN fans. Martin D when he originally came onto the project wanted to
>>use
>> Git, and was more familiar with it, but took great effort to adopt SVN
>>b/c
>> ASF support for Git at that time was quite limited.
>>
>> However, with you here now; with Adam; with Martin; and with a number of
>> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
>> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
>> release
>> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
>> move to Git if this discussion is favorable. Apache now supports
>>writeable
>> Git repos (see http://git.apache.org/) and the project's canonical
>> repository
>> can be Git. We can still mirror to Github, etc., but the bits (and
>>really
>> the
>> work) ought to be happening here at the ASF.
>>
>> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
I think that a migration to git would be desirable in medium term. But
on my side, I need to learn git more, in particular how to manage
branches with git. The DVS that I know was Mercurial actually.
SVN works reasonably well for now. However merging is not yet as
efficient than Mercurial (and presumably Git). In particular, giving the
same file copied on 2 branches:
branches/A/MyFile
branches/B/MyFile
If we rename MyFile on branch A, then merge with branch B, the changes
done in that file on branch B are lost with SVN, while they are
preserved with Mercurial. Until now I took that limitation as an
incitative to think harder about the module/package/class names in the
first place, and to not move lightly.
Before I could switch to git, I would need to learn how to perform the
work equivalent to Mercurial's "named branches". Git does not exactly
have the Mercurial concept of named branches, but have something else
providing similar functionality (I think). We also need to investigate
about what will happen to directories other than the standard "trunk",
"branches" and "tags". We have "data", "ip-review", "presentations" and
"site". I'm not sure if those directories are already on the Git clone.
In summary: migration to Git would requires some work (I think), so
maybe it could be a medium term goal, somewhere after the 0.3 release?
Martin
Le 20/06/13 16:37, Mattmann, Chris A (398J) a écrit :
> I have a simple proposal :) You guys are definitely more Git fans now than
> SVN fans. Martin D when he originally came onto the project wanted to use
> Git, and was more familiar with it, but took great effort to adopt SVN b/c
> ASF support for Git at that time was quite limited.
>
> However, with you here now; with Adam; with Martin; and with a number of
> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> release
> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> move to Git if this discussion is favorable. Apache now supports writeable
> Git repos (see http://git.apache.org/) and the project's canonical
> repository
> can be Git. We can still mirror to Github, etc., but the bits (and really
> the
> work) ought to be happening here at the ASF.
>
> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Travis L Pinney <tr...@gmail.com>.
Thanks!
On Thu, Jun 20, 2013 at 10:49 AM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Travis, +1 go for it (in terms of making the additional branch
> for now). They're cheap; you have commit karma; you are all good!
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Travis L Pinney <tr...@gmail.com>
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Thursday, June 20, 2013 7:41 AM
> To: dev <de...@sis.apache.org>
> Subject: Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
> branch)
>
> >+1 on git after 0.3
> >
> >+1 on apache hardware.
> >
> >For experimental processing (MapReduce and Shapefiles), should I make that
> >another svn branch for now?
> >
> >Thanks,
> >Travis
> >
> >
> >
> >
> >
> >
> >On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hey Travis,
> >>
> >> I would strongly urge you to do development on Apache SIS on Apache
> >> hardware.
> >> Github is great; and convenience. But when you commit there, we don't
> >>get
> >> email notifications and so forth here and the community loses out (and
> >>we
> >> lose
> >> out) on having email records; archives, and other things here that show
> >> work
> >> is going on in SIS.
> >>
> >> I have a simple proposal :) You guys are definitely more Git fans now
> >>than
> >> SVN fans. Martin D when he originally came onto the project wanted to
> >>use
> >> Git, and was more familiar with it, but took great effort to adopt SVN
> >>b/c
> >> ASF support for Git at that time was quite limited.
> >>
> >> However, with you here now; with Adam; with Martin; and with a number of
> >> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> >> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> >> release
> >> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> >> move to Git if this discussion is favorable. Apache now supports
> >>writeable
> >> Git repos (see http://git.apache.org/) and the project's canonical
> >> repository
> >> can be Git. We can still mirror to Github, etc., but the bits (and
> >>really
> >> the
> >> work) ought to be happening here at the ASF.
> >>
> >> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Travis L Pinney <tr...@gmail.com>
> >> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> >> Date: Thursday, June 20, 2013 7:31 AM
> >> To: dev <de...@sis.apache.org>
> >> Subject: Re: shapefile branch
> >>
> >> >Good to know about the OGC/ISO interfaces.
> >> >
> >> >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
> >> >files
> >> >etc. I can set up in another code repo on github. The reason I want to
> >> >work
> >> >on that concurrently is to stress test the existing library with lots
> >>of
> >> >data to find bugs that may not appear with simple unit tests.
> >> >
> >> >
> >> >
> >> >Thanks,
> >> >Travis
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
> >> >martin.desruisseaux@geomatys.fr> wrote:
> >> >
> >> >> Le 20/06/13 12:47, Travis L Pinney a écrit :
> >> >>
> >> >> The java.util.Map is fairly basic now. An improvement could be a
> >> >>feature
> >> >>> class that has a map of <String, DataType>, where DataType
> >>corresponds
> >> >>>to
> >> >>> the appropriate DataType (
> >> >>>
> >>
> >>>>>
> http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
> >> <h
> >> >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
> >> >>> .)
> >> >>> Currently I am converting everything to strings.
> >> >>>
> >> >>
> >> >> Actually Feature, FeatureType and related interfaces derived from
> >> >>OGC/ISO
> >> >> standards (in particular GML - Geographic Markup Language - schemas)
> >>are
> >> >> already provided in GeoAPI:
> >> >>
> >> >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
> >> >>
> >> >>feature/package-summary.html<
> >> http://www.geoapi.org/snapshot/pending/org/o
> >> >>pengis/feature/package-summary.html>
> >> >>
> >> >> This is in the "pending" part of GeoAPI, so we have room for revising
> >> >> them, in particular make sure that they are still in agreement with
> >> >>latest
> >> >> OGC/ISO standards. Then we would need to provide an implementation in
> >> >>SIS,
> >> >> porting Geotk classes when possible or appropriate. However there is
> >>a
> >> >> somewhat long road before we reach that point, so it seems to me that
> >> >>your
> >> >> current approach (String in java.util.Map) is good in the main time.
> >> >>
> >> >>
> >> >>
> >> >> The bulk ingests would be an api where you can call a jar file from
> >> >>> hadoop,
> >> >>> give it appropriate directory to pull shapefiles in HDFS, and it
> >>would
> >> >>> process each shapefile per mapper. The first ingest I am working on
> >>is
> >> >>>a
> >> >>> transformation of points to a 2D-histogram to get an idea of
> >>density of
> >> >>> features of all the shapefiles. This could be extended to have
> >> >>>different
> >> >>> types of outputs (store in a database or more efficient format on
> >>hdfs)
> >> >>>
> >> >>
> >> >> I would suggest to separate the two tasks. I think that the above is
> >> >>what
> >> >> we call a "processing", which is the subject of (yet an other) OGC
> >> >> standard. Processing and DataStore should be independent, i.e.
> >>someone
> >> >>may
> >> >> want to apply the above processing on NetCDF files too... Maybe we
> >>can
> >> >> focus on ShapefileStore first, and revisit processing later?
> >>Processings
> >> >> will need DataStores first in order to perform their work anyway...
> >> >>
> >> >> Martin
> >> >>
> >> >>
> >>
> >>
>
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Adam Estrada <es...@gmail.com>.
Yeah Yeah Yeah...me too ;-) +1 from me ;-)
On Thu, Jun 20, 2013 at 10:50 AM, Joe White <wh...@gmail.com> wrote:
> +1 on Git. I'm not a Git guy, but I like expanding the horizons some.
>
> Joe
> On Jun 20, 2013, at 10:41 AM, Travis L Pinney <tr...@gmail.com>
> wrote:
>
> > +1 on git after 0.3
> >
> > +1 on apache hardware.
> >
> > For experimental processing (MapReduce and Shapefiles), should I make
> that
> > another svn branch for now?
> >
> > Thanks,
> > Travis
> >
> >
> >
> >
> >
> >
> > On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hey Travis,
> >>
> >> I would strongly urge you to do development on Apache SIS on Apache
> >> hardware.
> >> Github is great; and convenience. But when you commit there, we don't
> get
> >> email notifications and so forth here and the community loses out (and
> we
> >> lose
> >> out) on having email records; archives, and other things here that show
> >> work
> >> is going on in SIS.
> >>
> >> I have a simple proposal :) You guys are definitely more Git fans now
> than
> >> SVN fans. Martin D when he originally came onto the project wanted to
> use
> >> Git, and was more familiar with it, but took great effort to adopt SVN
> b/c
> >> ASF support for Git at that time was quite limited.
> >>
> >> However, with you here now; with Adam; with Martin; and with a number of
> >> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> >> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> >> release
> >> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> >> move to Git if this discussion is favorable. Apache now supports
> writeable
> >> Git repos (see http://git.apache.org/) and the project's canonical
> >> repository
> >> can be Git. We can still mirror to Github, etc., but the bits (and
> really
> >> the
> >> work) ought to be happening here at the ASF.
> >>
> >> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Travis L Pinney <tr...@gmail.com>
> >> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> >> Date: Thursday, June 20, 2013 7:31 AM
> >> To: dev <de...@sis.apache.org>
> >> Subject: Re: shapefile branch
> >>
> >>> Good to know about the OGC/ISO interfaces.
> >>>
> >>> It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
> >>> files
> >>> etc. I can set up in another code repo on github. The reason I want to
> >>> work
> >>> on that concurrently is to stress test the existing library with lots
> of
> >>> data to find bugs that may not appear with simple unit tests.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> Travis
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
> >>> martin.desruisseaux@geomatys.fr> wrote:
> >>>
> >>>> Le 20/06/13 12:47, Travis L Pinney a écrit :
> >>>>
> >>>> The java.util.Map is fairly basic now. An improvement could be a
> >>>> feature
> >>>>> class that has a map of <String, DataType>, where DataType
> corresponds
> >>>>> to
> >>>>> the appropriate DataType (
> >>>>>
> >>>>>
> http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
> >> <h
> >>>>> ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
> >>>>> .)
> >>>>> Currently I am converting everything to strings.
> >>>>>
> >>>>
> >>>> Actually Feature, FeatureType and related interfaces derived from
> >>>> OGC/ISO
> >>>> standards (in particular GML - Geographic Markup Language - schemas)
> are
> >>>> already provided in GeoAPI:
> >>>>
> >>>> http://www.geoapi.org/**snapshot/pending/org/opengis/**
> >>>>
> >>>> feature/package-summary.html<
> >> http://www.geoapi.org/snapshot/pending/org/o
> >>>> pengis/feature/package-summary.html>
> >>>>
> >>>> This is in the "pending" part of GeoAPI, so we have room for revising
> >>>> them, in particular make sure that they are still in agreement with
> >>>> latest
> >>>> OGC/ISO standards. Then we would need to provide an implementation in
> >>>> SIS,
> >>>> porting Geotk classes when possible or appropriate. However there is a
> >>>> somewhat long road before we reach that point, so it seems to me that
> >>>> your
> >>>> current approach (String in java.util.Map) is good in the main time.
> >>>>
> >>>>
> >>>>
> >>>> The bulk ingests would be an api where you can call a jar file from
> >>>>> hadoop,
> >>>>> give it appropriate directory to pull shapefiles in HDFS, and it
> would
> >>>>> process each shapefile per mapper. The first ingest I am working on
> is
> >>>>> a
> >>>>> transformation of points to a 2D-histogram to get an idea of density
> of
> >>>>> features of all the shapefiles. This could be extended to have
> >>>>> different
> >>>>> types of outputs (store in a database or more efficient format on
> hdfs)
> >>>>>
> >>>>
> >>>> I would suggest to separate the two tasks. I think that the above is
> >>>> what
> >>>> we call a "processing", which is the subject of (yet an other) OGC
> >>>> standard. Processing and DataStore should be independent, i.e. someone
> >>>> may
> >>>> want to apply the above processing on NetCDF files too... Maybe we can
> >>>> focus on ShapefileStore first, and revisit processing later?
> Processings
> >>>> will need DataStores first in order to perform their work anyway...
> >>>>
> >>>> Martin
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Joe White <wh...@gmail.com>.
+1 on Git. I'm not a Git guy, but I like expanding the horizons some.
Joe
On Jun 20, 2013, at 10:41 AM, Travis L Pinney <tr...@gmail.com> wrote:
> +1 on git after 0.3
>
> +1 on apache hardware.
>
> For experimental processing (MapReduce and Shapefiles), should I make that
> another svn branch for now?
>
> Thanks,
> Travis
>
>
>
>
>
>
> On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hey Travis,
>>
>> I would strongly urge you to do development on Apache SIS on Apache
>> hardware.
>> Github is great; and convenience. But when you commit there, we don't get
>> email notifications and so forth here and the community loses out (and we
>> lose
>> out) on having email records; archives, and other things here that show
>> work
>> is going on in SIS.
>>
>> I have a simple proposal :) You guys are definitely more Git fans now than
>> SVN fans. Martin D when he originally came onto the project wanted to use
>> Git, and was more familiar with it, but took great effort to adopt SVN b/c
>> ASF support for Git at that time was quite limited.
>>
>> However, with you here now; with Adam; with Martin; and with a number of
>> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
>> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
>> release
>> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
>> move to Git if this discussion is favorable. Apache now supports writeable
>> Git repos (see http://git.apache.org/) and the project's canonical
>> repository
>> can be Git. We can still mirror to Github, etc., but the bits (and really
>> the
>> work) ought to be happening here at the ASF.
>>
>> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Travis L Pinney <tr...@gmail.com>
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Thursday, June 20, 2013 7:31 AM
>> To: dev <de...@sis.apache.org>
>> Subject: Re: shapefile branch
>>
>>> Good to know about the OGC/ISO interfaces.
>>>
>>> It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
>>> files
>>> etc. I can set up in another code repo on github. The reason I want to
>>> work
>>> on that concurrently is to stress test the existing library with lots of
>>> data to find bugs that may not appear with simple unit tests.
>>>
>>>
>>>
>>> Thanks,
>>> Travis
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
>>> martin.desruisseaux@geomatys.fr> wrote:
>>>
>>>> Le 20/06/13 12:47, Travis L Pinney a écrit :
>>>>
>>>> The java.util.Map is fairly basic now. An improvement could be a
>>>> feature
>>>>> class that has a map of <String, DataType>, where DataType corresponds
>>>>> to
>>>>> the appropriate DataType (
>>>>>
>>>>> http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
>> <h
>>>>> ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
>>>>> .)
>>>>> Currently I am converting everything to strings.
>>>>>
>>>>
>>>> Actually Feature, FeatureType and related interfaces derived from
>>>> OGC/ISO
>>>> standards (in particular GML - Geographic Markup Language - schemas) are
>>>> already provided in GeoAPI:
>>>>
>>>> http://www.geoapi.org/**snapshot/pending/org/opengis/**
>>>>
>>>> feature/package-summary.html<
>> http://www.geoapi.org/snapshot/pending/org/o
>>>> pengis/feature/package-summary.html>
>>>>
>>>> This is in the "pending" part of GeoAPI, so we have room for revising
>>>> them, in particular make sure that they are still in agreement with
>>>> latest
>>>> OGC/ISO standards. Then we would need to provide an implementation in
>>>> SIS,
>>>> porting Geotk classes when possible or appropriate. However there is a
>>>> somewhat long road before we reach that point, so it seems to me that
>>>> your
>>>> current approach (String in java.util.Map) is good in the main time.
>>>>
>>>>
>>>>
>>>> The bulk ingests would be an api where you can call a jar file from
>>>>> hadoop,
>>>>> give it appropriate directory to pull shapefiles in HDFS, and it would
>>>>> process each shapefile per mapper. The first ingest I am working on is
>>>>> a
>>>>> transformation of points to a 2D-histogram to get an idea of density of
>>>>> features of all the shapefiles. This could be extended to have
>>>>> different
>>>>> types of outputs (store in a database or more efficient format on hdfs)
>>>>>
>>>>
>>>> I would suggest to separate the two tasks. I think that the above is
>>>> what
>>>> we call a "processing", which is the subject of (yet an other) OGC
>>>> standard. Processing and DataStore should be independent, i.e. someone
>>>> may
>>>> want to apply the above processing on NetCDF files too... Maybe we can
>>>> focus on ShapefileStore first, and revisit processing later? Processings
>>>> will need DataStores first in order to perform their work anyway...
>>>>
>>>> Martin
>>>>
>>>>
>>
>>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Travis, +1 go for it (in terms of making the additional branch
for now). They're cheap; you have commit karma; you are all good!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Thursday, June 20, 2013 7:41 AM
To: dev <de...@sis.apache.org>
Subject: Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
>+1 on git after 0.3
>
>+1 on apache hardware.
>
>For experimental processing (MapReduce and Shapefiles), should I make that
>another svn branch for now?
>
>Thanks,
>Travis
>
>
>
>
>
>
>On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hey Travis,
>>
>> I would strongly urge you to do development on Apache SIS on Apache
>> hardware.
>> Github is great; and convenience. But when you commit there, we don't
>>get
>> email notifications and so forth here and the community loses out (and
>>we
>> lose
>> out) on having email records; archives, and other things here that show
>> work
>> is going on in SIS.
>>
>> I have a simple proposal :) You guys are definitely more Git fans now
>>than
>> SVN fans. Martin D when he originally came onto the project wanted to
>>use
>> Git, and was more familiar with it, but took great effort to adopt SVN
>>b/c
>> ASF support for Git at that time was quite limited.
>>
>> However, with you here now; with Adam; with Martin; and with a number of
>> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
>> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
>> release
>> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
>> move to Git if this discussion is favorable. Apache now supports
>>writeable
>> Git repos (see http://git.apache.org/) and the project's canonical
>> repository
>> can be Git. We can still mirror to Github, etc., but the bits (and
>>really
>> the
>> work) ought to be happening here at the ASF.
>>
>> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Travis L Pinney <tr...@gmail.com>
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Thursday, June 20, 2013 7:31 AM
>> To: dev <de...@sis.apache.org>
>> Subject: Re: shapefile branch
>>
>> >Good to know about the OGC/ISO interfaces.
>> >
>> >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
>> >files
>> >etc. I can set up in another code repo on github. The reason I want to
>> >work
>> >on that concurrently is to stress test the existing library with lots
>>of
>> >data to find bugs that may not appear with simple unit tests.
>> >
>> >
>> >
>> >Thanks,
>> >Travis
>> >
>> >
>> >
>> >
>> >
>> >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
>> >martin.desruisseaux@geomatys.fr> wrote:
>> >
>> >> Le 20/06/13 12:47, Travis L Pinney a écrit :
>> >>
>> >> The java.util.Map is fairly basic now. An improvement could be a
>> >>feature
>> >>> class that has a map of <String, DataType>, where DataType
>>corresponds
>> >>>to
>> >>> the appropriate DataType (
>> >>>
>>
>>>>>http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
>> <h
>> >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
>> >>> .)
>> >>> Currently I am converting everything to strings.
>> >>>
>> >>
>> >> Actually Feature, FeatureType and related interfaces derived from
>> >>OGC/ISO
>> >> standards (in particular GML - Geographic Markup Language - schemas)
>>are
>> >> already provided in GeoAPI:
>> >>
>> >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
>> >>
>> >>feature/package-summary.html<
>> http://www.geoapi.org/snapshot/pending/org/o
>> >>pengis/feature/package-summary.html>
>> >>
>> >> This is in the "pending" part of GeoAPI, so we have room for revising
>> >> them, in particular make sure that they are still in agreement with
>> >>latest
>> >> OGC/ISO standards. Then we would need to provide an implementation in
>> >>SIS,
>> >> porting Geotk classes when possible or appropriate. However there is
>>a
>> >> somewhat long road before we reach that point, so it seems to me that
>> >>your
>> >> current approach (String in java.util.Map) is good in the main time.
>> >>
>> >>
>> >>
>> >> The bulk ingests would be an api where you can call a jar file from
>> >>> hadoop,
>> >>> give it appropriate directory to pull shapefiles in HDFS, and it
>>would
>> >>> process each shapefile per mapper. The first ingest I am working on
>>is
>> >>>a
>> >>> transformation of points to a 2D-histogram to get an idea of
>>density of
>> >>> features of all the shapefiles. This could be extended to have
>> >>>different
>> >>> types of outputs (store in a database or more efficient format on
>>hdfs)
>> >>>
>> >>
>> >> I would suggest to separate the two tasks. I think that the above is
>> >>what
>> >> we call a "processing", which is the subject of (yet an other) OGC
>> >> standard. Processing and DataStore should be independent, i.e.
>>someone
>> >>may
>> >> want to apply the above processing on NetCDF files too... Maybe we
>>can
>> >> focus on ShapefileStore first, and revisit processing later?
>>Processings
>> >> will need DataStores first in order to perform their work anyway...
>> >>
>> >> Martin
>> >>
>> >>
>>
>>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Travis L Pinney <tr...@gmail.com>.
+1 on git after 0.3
+1 on apache hardware.
For experimental processing (MapReduce and Shapefiles), should I make that
another svn branch for now?
Thanks,
Travis
On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Travis,
>
> I would strongly urge you to do development on Apache SIS on Apache
> hardware.
> Github is great; and convenience. But when you commit there, we don't get
> email notifications and so forth here and the community loses out (and we
> lose
> out) on having email records; archives, and other things here that show
> work
> is going on in SIS.
>
> I have a simple proposal :) You guys are definitely more Git fans now than
> SVN fans. Martin D when he originally came onto the project wanted to use
> Git, and was more familiar with it, but took great effort to adopt SVN b/c
> ASF support for Git at that time was quite limited.
>
> However, with you here now; with Adam; with Martin; and with a number of
> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> release
> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> move to Git if this discussion is favorable. Apache now supports writeable
> Git repos (see http://git.apache.org/) and the project's canonical
> repository
> can be Git. We can still mirror to Github, etc., but the bits (and really
> the
> work) ought to be happening here at the ASF.
>
> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Travis L Pinney <tr...@gmail.com>
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Thursday, June 20, 2013 7:31 AM
> To: dev <de...@sis.apache.org>
> Subject: Re: shapefile branch
>
> >Good to know about the OGC/ISO interfaces.
> >
> >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
> >files
> >etc. I can set up in another code repo on github. The reason I want to
> >work
> >on that concurrently is to stress test the existing library with lots of
> >data to find bugs that may not appear with simple unit tests.
> >
> >
> >
> >Thanks,
> >Travis
> >
> >
> >
> >
> >
> >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
> >martin.desruisseaux@geomatys.fr> wrote:
> >
> >> Le 20/06/13 12:47, Travis L Pinney a écrit :
> >>
> >> The java.util.Map is fairly basic now. An improvement could be a
> >>feature
> >>> class that has a map of <String, DataType>, where DataType corresponds
> >>>to
> >>> the appropriate DataType (
> >>>
> >>>http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
> <h
> >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
> >>> .)
> >>> Currently I am converting everything to strings.
> >>>
> >>
> >> Actually Feature, FeatureType and related interfaces derived from
> >>OGC/ISO
> >> standards (in particular GML - Geographic Markup Language - schemas) are
> >> already provided in GeoAPI:
> >>
> >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
> >>
> >>feature/package-summary.html<
> http://www.geoapi.org/snapshot/pending/org/o
> >>pengis/feature/package-summary.html>
> >>
> >> This is in the "pending" part of GeoAPI, so we have room for revising
> >> them, in particular make sure that they are still in agreement with
> >>latest
> >> OGC/ISO standards. Then we would need to provide an implementation in
> >>SIS,
> >> porting Geotk classes when possible or appropriate. However there is a
> >> somewhat long road before we reach that point, so it seems to me that
> >>your
> >> current approach (String in java.util.Map) is good in the main time.
> >>
> >>
> >>
> >> The bulk ingests would be an api where you can call a jar file from
> >>> hadoop,
> >>> give it appropriate directory to pull shapefiles in HDFS, and it would
> >>> process each shapefile per mapper. The first ingest I am working on is
> >>>a
> >>> transformation of points to a 2D-histogram to get an idea of density of
> >>> features of all the shapefiles. This could be extended to have
> >>>different
> >>> types of outputs (store in a database or more efficient format on hdfs)
> >>>
> >>
> >> I would suggest to separate the two tasks. I think that the above is
> >>what
> >> we call a "processing", which is the subject of (yet an other) OGC
> >> standard. Processing and DataStore should be independent, i.e. someone
> >>may
> >> want to apply the above processing on NetCDF files too... Maybe we can
> >> focus on ShapefileStore first, and revisit processing later? Processings
> >> will need DataStores first in order to perform their work anyway...
> >>
> >> Martin
> >>
> >>
>
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Travis L Pinney <tr...@gmail.com>.
It should be possible.
At the very least you should be able to check out a svn clone from any
github repo
svn co https://github.com/apache/sis
On Thu, Jun 20, 2013 at 10:45 AM, Adam Estrada <es...@gmail.com>wrote:
> I really don't mind using Git either as long as SVN doesn't completely go
> away. I am more of a SVN fan anyway...Are we able to run them at the same
> time in a mirrored mode?
>
>
> On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hey Travis,
> >
> > I would strongly urge you to do development on Apache SIS on Apache
> > hardware.
> > Github is great; and convenience. But when you commit there, we don't get
> > email notifications and so forth here and the community loses out (and we
> > lose
> > out) on having email records; archives, and other things here that show
> > work
> > is going on in SIS.
> >
> > I have a simple proposal :) You guys are definitely more Git fans now
> than
> > SVN fans. Martin D when he originally came onto the project wanted to use
> > Git, and was more familiar with it, but took great effort to adopt SVN
> b/c
> > ASF support for Git at that time was quite limited.
> >
> > However, with you here now; with Adam; with Martin; and with a number of
> > other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> > it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> > release
> > that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> > move to Git if this discussion is favorable. Apache now supports
> writeable
> > Git repos (see http://git.apache.org/) and the project's canonical
> > repository
> > can be Git. We can still mirror to Github, etc., but the bits (and really
> > the
> > work) ought to be happening here at the ASF.
> >
> > So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
> >
> > Cheers,
> > Chris
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Travis L Pinney <tr...@gmail.com>
> > Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> > Date: Thursday, June 20, 2013 7:31 AM
> > To: dev <de...@sis.apache.org>
> > Subject: Re: shapefile branch
> >
> > >Good to know about the OGC/ISO interfaces.
> > >
> > >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
> > >files
> > >etc. I can set up in another code repo on github. The reason I want to
> > >work
> > >on that concurrently is to stress test the existing library with lots of
> > >data to find bugs that may not appear with simple unit tests.
> > >
> > >
> > >
> > >Thanks,
> > >Travis
> > >
> > >
> > >
> > >
> > >
> > >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
> > >martin.desruisseaux@geomatys.fr> wrote:
> > >
> > >> Le 20/06/13 12:47, Travis L Pinney a écrit :
> > >>
> > >> The java.util.Map is fairly basic now. An improvement could be a
> > >>feature
> > >>> class that has a map of <String, DataType>, where DataType
> corresponds
> > >>>to
> > >>> the appropriate DataType (
> > >>>
> > >>>
> http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
> > <h
> > >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
> > >>> .)
> > >>> Currently I am converting everything to strings.
> > >>>
> > >>
> > >> Actually Feature, FeatureType and related interfaces derived from
> > >>OGC/ISO
> > >> standards (in particular GML - Geographic Markup Language - schemas)
> are
> > >> already provided in GeoAPI:
> > >>
> > >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
> > >>
> > >>feature/package-summary.html<
> > http://www.geoapi.org/snapshot/pending/org/o
> > >>pengis/feature/package-summary.html>
> > >>
> > >> This is in the "pending" part of GeoAPI, so we have room for revising
> > >> them, in particular make sure that they are still in agreement with
> > >>latest
> > >> OGC/ISO standards. Then we would need to provide an implementation in
> > >>SIS,
> > >> porting Geotk classes when possible or appropriate. However there is a
> > >> somewhat long road before we reach that point, so it seems to me that
> > >>your
> > >> current approach (String in java.util.Map) is good in the main time.
> > >>
> > >>
> > >>
> > >> The bulk ingests would be an api where you can call a jar file from
> > >>> hadoop,
> > >>> give it appropriate directory to pull shapefiles in HDFS, and it
> would
> > >>> process each shapefile per mapper. The first ingest I am working on
> is
> > >>>a
> > >>> transformation of points to a 2D-histogram to get an idea of density
> of
> > >>> features of all the shapefiles. This could be extended to have
> > >>>different
> > >>> types of outputs (store in a database or more efficient format on
> hdfs)
> > >>>
> > >>
> > >> I would suggest to separate the two tasks. I think that the above is
> > >>what
> > >> we call a "processing", which is the subject of (yet an other) OGC
> > >> standard. Processing and DataStore should be independent, i.e. someone
> > >>may
> > >> want to apply the above processing on NetCDF files too... Maybe we can
> > >> focus on ShapefileStore first, and revisit processing later?
> Processings
> > >> will need DataStores first in order to perform their work anyway...
> > >>
> > >> Martin
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Hey Adam,
It's really one or the other unfortunately -- infra doesn't have
the time or resources to keep them in sync (and doesn't really make
sense anyways).
I brought it mostly just b/c I thought it might ease the pension
for folks to go do things at Github when I think they should be
doing them here at the ASF.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Adam Estrada <es...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Thursday, June 20, 2013 7:45 AM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile
branch)
>I really don't mind using Git either as long as SVN doesn't completely go
>away. I am more of a SVN fan anyway...Are we able to run them at the same
>time in a mirrored mode?
>
>
>On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hey Travis,
>>
>> I would strongly urge you to do development on Apache SIS on Apache
>> hardware.
>> Github is great; and convenience. But when you commit there, we don't
>>get
>> email notifications and so forth here and the community loses out (and
>>we
>> lose
>> out) on having email records; archives, and other things here that show
>> work
>> is going on in SIS.
>>
>> I have a simple proposal :) You guys are definitely more Git fans now
>>than
>> SVN fans. Martin D when he originally came onto the project wanted to
>>use
>> Git, and was more familiar with it, but took great effort to adopt SVN
>>b/c
>> ASF support for Git at that time was quite limited.
>>
>> However, with you here now; with Adam; with Martin; and with a number of
>> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
>> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
>> release
>> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
>> move to Git if this discussion is favorable. Apache now supports
>>writeable
>> Git repos (see http://git.apache.org/) and the project's canonical
>> repository
>> can be Git. We can still mirror to Github, etc., but the bits (and
>>really
>> the
>> work) ought to be happening here at the ASF.
>>
>> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Travis L Pinney <tr...@gmail.com>
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Thursday, June 20, 2013 7:31 AM
>> To: dev <de...@sis.apache.org>
>> Subject: Re: shapefile branch
>>
>> >Good to know about the OGC/ISO interfaces.
>> >
>> >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
>> >files
>> >etc. I can set up in another code repo on github. The reason I want to
>> >work
>> >on that concurrently is to stress test the existing library with lots
>>of
>> >data to find bugs that may not appear with simple unit tests.
>> >
>> >
>> >
>> >Thanks,
>> >Travis
>> >
>> >
>> >
>> >
>> >
>> >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
>> >martin.desruisseaux@geomatys.fr> wrote:
>> >
>> >> Le 20/06/13 12:47, Travis L Pinney a écrit :
>> >>
>> >> The java.util.Map is fairly basic now. An improvement could be a
>> >>feature
>> >>> class that has a map of <String, DataType>, where DataType
>>corresponds
>> >>>to
>> >>> the appropriate DataType (
>> >>>
>>
>>>>>http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
>> <h
>> >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
>> >>> .)
>> >>> Currently I am converting everything to strings.
>> >>>
>> >>
>> >> Actually Feature, FeatureType and related interfaces derived from
>> >>OGC/ISO
>> >> standards (in particular GML - Geographic Markup Language - schemas)
>>are
>> >> already provided in GeoAPI:
>> >>
>> >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
>> >>
>> >>feature/package-summary.html<
>> http://www.geoapi.org/snapshot/pending/org/o
>> >>pengis/feature/package-summary.html>
>> >>
>> >> This is in the "pending" part of GeoAPI, so we have room for revising
>> >> them, in particular make sure that they are still in agreement with
>> >>latest
>> >> OGC/ISO standards. Then we would need to provide an implementation in
>> >>SIS,
>> >> porting Geotk classes when possible or appropriate. However there is
>>a
>> >> somewhat long road before we reach that point, so it seems to me that
>> >>your
>> >> current approach (String in java.util.Map) is good in the main time.
>> >>
>> >>
>> >>
>> >> The bulk ingests would be an api where you can call a jar file from
>> >>> hadoop,
>> >>> give it appropriate directory to pull shapefiles in HDFS, and it
>>would
>> >>> process each shapefile per mapper. The first ingest I am working on
>>is
>> >>>a
>> >>> transformation of points to a 2D-histogram to get an idea of
>>density of
>> >>> features of all the shapefiles. This could be extended to have
>> >>>different
>> >>> types of outputs (store in a database or more efficient format on
>>hdfs)
>> >>>
>> >>
>> >> I would suggest to separate the two tasks. I think that the above is
>> >>what
>> >> we call a "processing", which is the subject of (yet an other) OGC
>> >> standard. Processing and DataStore should be independent, i.e.
>>someone
>> >>may
>> >> want to apply the above processing on NetCDF files too... Maybe we
>>can
>> >> focus on ShapefileStore first, and revisit processing later?
>>Processings
>> >> will need DataStores first in order to perform their work anyway...
>> >>
>> >> Martin
>> >>
>> >>
>>
>>
Re: [DISCUSS] Transition to Git after 0.3? (was Re: shapefile branch)
Posted by Adam Estrada <es...@gmail.com>.
I really don't mind using Git either as long as SVN doesn't completely go
away. I am more of a SVN fan anyway...Are we able to run them at the same
time in a mirrored mode?
On Thu, Jun 20, 2013 at 10:37 AM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Travis,
>
> I would strongly urge you to do development on Apache SIS on Apache
> hardware.
> Github is great; and convenience. But when you commit there, we don't get
> email notifications and so forth here and the community loses out (and we
> lose
> out) on having email records; archives, and other things here that show
> work
> is going on in SIS.
>
> I have a simple proposal :) You guys are definitely more Git fans now than
> SVN fans. Martin D when he originally came onto the project wanted to use
> Git, and was more familiar with it, but took great effort to adopt SVN b/c
> ASF support for Git at that time was quite limited.
>
> However, with you here now; with Adam; with Martin; and with a number of
> other folks contributing (Joe W. are you a Git guy?) that are Git fans,
> it's worth revisiting this discussion. However, *after* 0.3 :) Let's
> release
> that using SVN so we don't hold that off anymore. After 0.3 maybe we can
> move to Git if this discussion is favorable. Apache now supports writeable
> Git repos (see http://git.apache.org/) and the project's canonical
> repository
> can be Git. We can still mirror to Github, etc., but the bits (and really
> the
> work) ought to be happening here at the ASF.
>
> So, discuss please :) FWIW, I'm +1 to move to Git (after 0.3).
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Travis L Pinney <tr...@gmail.com>
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Thursday, June 20, 2013 7:31 AM
> To: dev <de...@sis.apache.org>
> Subject: Re: shapefile branch
>
> >Good to know about the OGC/ISO interfaces.
> >
> >It would make sense to apply processing to NetCDF, Shapefile, Mbtiles
> >files
> >etc. I can set up in another code repo on github. The reason I want to
> >work
> >on that concurrently is to stress test the existing library with lots of
> >data to find bugs that may not appear with simple unit tests.
> >
> >
> >
> >Thanks,
> >Travis
> >
> >
> >
> >
> >
> >On Thu, Jun 20, 2013 at 7:42 AM, Martin Desruisseaux <
> >martin.desruisseaux@geomatys.fr> wrote:
> >
> >> Le 20/06/13 12:47, Travis L Pinney a écrit :
> >>
> >> The java.util.Map is fairly basic now. An improvement could be a
> >>feature
> >>> class that has a map of <String, DataType>, where DataType corresponds
> >>>to
> >>> the appropriate DataType (
> >>>
> >>>http://www.clicketyclick.dk/**databases/xbase/format/data_**types.html
> <h
> >>>ttp://www.clicketyclick.dk/databases/xbase/format/data_types.html>
> >>> .)
> >>> Currently I am converting everything to strings.
> >>>
> >>
> >> Actually Feature, FeatureType and related interfaces derived from
> >>OGC/ISO
> >> standards (in particular GML - Geographic Markup Language - schemas) are
> >> already provided in GeoAPI:
> >>
> >> http://www.geoapi.org/**snapshot/pending/org/opengis/**
> >>
> >>feature/package-summary.html<
> http://www.geoapi.org/snapshot/pending/org/o
> >>pengis/feature/package-summary.html>
> >>
> >> This is in the "pending" part of GeoAPI, so we have room for revising
> >> them, in particular make sure that they are still in agreement with
> >>latest
> >> OGC/ISO standards. Then we would need to provide an implementation in
> >>SIS,
> >> porting Geotk classes when possible or appropriate. However there is a
> >> somewhat long road before we reach that point, so it seems to me that
> >>your
> >> current approach (String in java.util.Map) is good in the main time.
> >>
> >>
> >>
> >> The bulk ingests would be an api where you can call a jar file from
> >>> hadoop,
> >>> give it appropriate directory to pull shapefiles in HDFS, and it would
> >>> process each shapefile per mapper. The first ingest I am working on is
> >>>a
> >>> transformation of points to a 2D-histogram to get an idea of density of
> >>> features of all the shapefiles. This could be extended to have
> >>>different
> >>> types of outputs (store in a database or more efficient format on hdfs)
> >>>
> >>
> >> I would suggest to separate the two tasks. I think that the above is
> >>what
> >> we call a "processing", which is the subject of (yet an other) OGC
> >> standard. Processing and DataStore should be independent, i.e. someone
> >>may
> >> want to apply the above processing on NetCDF files too... Maybe we can
> >> focus on ShapefileStore first, and revisit processing later? Processings
> >> will need DataStores first in order to perform their work anyway...
> >>
> >> Martin
> >>
> >>
>
>