You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by Travis L Pinney <tr...@gmail.com> on 2013/06/03 01:47:04 UTC
Re: Material for potential volunteer
Hello everyone,
I started working on a very rough prototype that can read in Shapefiles.
https://github.com/tlpinney/shapefile-api
In order to write a patch to submit, where would this component reside in
the Apache SIS project?
Thanks,
Travis
On Wed, May 8, 2013 at 3:11 AM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Sounds like a great opportunity for someone to step up here :)
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Martin Desruisseaux <ma...@geomatys.fr>
> Organization: Geomatys
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Tuesday, May 7, 2013 8:00 AM
> To: Apache SIS <de...@sis.apache.org>
> Subject: Material for potential volunteer
>
> >Hello all
> >
> >We filled an "Implement Shapefile data store" JIRA task as an idea for
> >potential volunteer. Actually it may be a bit early for implementing
> >such data store since we don't have yet any DataStore interface, no
> >Feature implementation, no Geometry and no CoordinateReferenceSystem.
> >Nevertheless a draft based on java.util.Map may be possible if some
> >volunteer wishes to put his hands in.
> >
> >https://issues.apache.org/jira/browse/SIS-100
> >
> >We filled this JIRA task because we can not port the Shapefile datastore
> >from Geotk for licensing reason (we can port many other datastores, but
> >not that one). So ideally, it would be better to have someone else from
> >the community to write at least a first draft. Johann Sorel, myself or
> >others could continue after that point, building on the first draft.
> >
> >No rush however. If it takes some time before such work begins, it may
> >leaves us the time to provide some of the missing bases.
> >
> > Martin
> >
>
>
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks Travis!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Tuesday, June 4, 2013 8:06 PM
To: dev <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Hi Chris,
>
>I went over your review board comments and I have everything checked in
>here.
>
>https://github.com/tlpinney/sis-shapefile/
>
>I still need to finish up the codepage support and the alternate reader.
>Once I have that I will make a patch and resubmit to review board.
>
>
>Thanks!
>Travis
>
>
>
>
>
>On Tue, Jun 4, 2013 at 6:28 PM, Mattmann, Chris A (398J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Thanks Martin cool!
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Martin Desruisseaux <ma...@geomatys.fr>
>> Organization: Geomatys
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Tuesday, June 4, 2013 2:46 PM
>> To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Subject: Re: Material for potential volunteer
>>
>> >Hello Chris
>> >
>> >Le 04/06/13 19:14, Mattmann, Chris A (398J) a écrit :
>> >> Feel free to suggest the same below in Review Board where you can
>> >> line by line do the comments below (and then in turn they will get
>> >> emailed to the mailing list).
>> >
>> >Thanks, I will do that on the next round.
>> >
>> > Cheers,
>> >
>> > Martin
>> >
>>
>>
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
Hi Chris,
I went over your review board comments and I have everything checked in
here.
https://github.com/tlpinney/sis-shapefile/
I still need to finish up the codepage support and the alternate reader.
Once I have that I will make a patch and resubmit to review board.
Thanks!
Travis
On Tue, Jun 4, 2013 at 6:28 PM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Thanks Martin cool!
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Martin Desruisseaux <ma...@geomatys.fr>
> Organization: Geomatys
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Tuesday, June 4, 2013 2:46 PM
> To: "dev@sis.apache.org" <de...@sis.apache.org>
> Subject: Re: Material for potential volunteer
>
> >Hello Chris
> >
> >Le 04/06/13 19:14, Mattmann, Chris A (398J) a écrit :
> >> Feel free to suggest the same below in Review Board where you can
> >> line by line do the comments below (and then in turn they will get
> >> emailed to the mailing list).
> >
> >Thanks, I will do that on the next round.
> >
> > Cheers,
> >
> > Martin
> >
>
>
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks Martin cool!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Tuesday, June 4, 2013 2:46 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Hello Chris
>
>Le 04/06/13 19:14, Mattmann, Chris A (398J) a écrit :
>> Feel free to suggest the same below in Review Board where you can
>> line by line do the comments below (and then in turn they will get
>> emailed to the mailing list).
>
>Thanks, I will do that on the next round.
>
> Cheers,
>
> Martin
>
Re: Material for potential volunteer
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Hello Chris
Le 04/06/13 19:14, Mattmann, Chris A (398J) a écrit :
> Feel free to suggest the same below in Review Board where you can
> line by line do the comments below (and then in turn they will get
> emailed to the mailing list).
Thanks, I will do that on the next round.
Cheers,
Martin
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Great, Martin.
Feel free to suggest the same below in Review Board where you can
line by line do the comments below (and then in turn they will get
emailed to the mailing list).
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Desruisseaux <ma...@geomatys.fr>
Organization: Geomatys
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Tuesday, June 4, 2013 2:05 AM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Hello Travis
>
>Le 04/06/13 04:40, Travis L Pinney a écrit :
>> Uploaded the patch here.
>>
>> https://reviews.apache.org/r/11615/
>
>Thanks! Can I suggest a few updates?
>
>It would be nice to have a little bit of javadoc, if possible with links
>to the Shapefile specification or wikipedia page related to the code.
>For example in ShapeTypeEnum, we don't know what the 'value' integer is...
>
>On a minor note, lines like below:
>
> StringBuilder s = new StringBuilder();
> s.append("FileCode: " + this.FileCode + "\n");
>
>Would be more efficient like below:
>
> StringBuilder s = new StringBuilder();
> s.append("FileCode: ").append(this.FileCode).append('\n');
>
>You may also consider replacing '\n' by lineSeparator, where:
>
> String lineSeparator = System.getProperty("line.separator", "\n");
>
>I noticed that many field name begins with an upper-case letter, while
>the Java convention is to start with lower-case letter. Could it be
>adjusted?
>
>In the following line:
>
> return new String(this.FieldName);
>
>The encoding need to be specified. I would expect the encoding to be
>specified somewhere in the Shapefile specification. For example if the
>encoding is ISO Latin 1, then the line should be:
>
> return new String(this.FieldName, "ISO-8859-1");
>
>I would also suggest to omit the "this." prefix, unless there is a risk
>of confusion with local variables.
>
>
>I would also suggest to avoid using MappedByteBuffer, and use plain
>ByteBuffer/ReadableChannel pair instead. The reason is that
>MappedByteBuffer is potentially heavy, and is usually reserved for very
>large file which are going to be open for a relatively long time.
>MappedByteBuffer are especially useful when bytes are going to be read
>in random order. For example I find MappedByteBuffer ideally suited for
>index.
>
>However in this case, we are reading data sequentially. It would be nice
>to avoid taking more OS resources than needed. Especially since the
>current Shapefile code creates 2 MappedByteBuffer and I suspect that one
>of them is for a small file. I realize that a ByteBuffer/ReadableChannel
>pair is slightly more tedious to use. But in order to make the task
>easier, you may if you want use the following convenience class,
>provided in the sis-storage module:
>
>https://svn.apache.org/repos/asf/sis/trunk/storage/sis-storage/src/main/ja
>va/org/apache/sis/internal/storage/ChannelDataInput.java
>
>Basically, you can give the ByteBuffer / ReadableChannel pair to the
>constructor and either:
>
> * Invoke ensureBufferContains(int) for specifying how many bytes you
> are going to need for the next sequence of ByteBuffer.get*()
>operations;
> * or use the convenience read*() methods when the number of bytes
> needed is not known in advance.
>
>
> Regards,
>
> Martin
>
Re: Material for potential volunteer
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Hello Travis
Le 04/06/13 04:40, Travis L Pinney a écrit :
> Uploaded the patch here.
>
> https://reviews.apache.org/r/11615/
Thanks! Can I suggest a few updates?
It would be nice to have a little bit of javadoc, if possible with links
to the Shapefile specification or wikipedia page related to the code.
For example in ShapeTypeEnum, we don't know what the 'value' integer is...
On a minor note, lines like below:
StringBuilder s = new StringBuilder();
s.append("FileCode: " + this.FileCode + "\n");
Would be more efficient like below:
StringBuilder s = new StringBuilder();
s.append("FileCode: ").append(this.FileCode).append('\n');
You may also consider replacing '\n' by lineSeparator, where:
String lineSeparator = System.getProperty("line.separator", "\n");
I noticed that many field name begins with an upper-case letter, while
the Java convention is to start with lower-case letter. Could it be
adjusted?
In the following line:
return new String(this.FieldName);
The encoding need to be specified. I would expect the encoding to be
specified somewhere in the Shapefile specification. For example if the
encoding is ISO Latin 1, then the line should be:
return new String(this.FieldName, "ISO-8859-1");
I would also suggest to omit the "this." prefix, unless there is a risk
of confusion with local variables.
I would also suggest to avoid using MappedByteBuffer, and use plain
ByteBuffer/ReadableChannel pair instead. The reason is that
MappedByteBuffer is potentially heavy, and is usually reserved for very
large file which are going to be open for a relatively long time.
MappedByteBuffer are especially useful when bytes are going to be read
in random order. For example I find MappedByteBuffer ideally suited for
index.
However in this case, we are reading data sequentially. It would be nice
to avoid taking more OS resources than needed. Especially since the
current Shapefile code creates 2 MappedByteBuffer and I suspect that one
of them is for a small file. I realize that a ByteBuffer/ReadableChannel
pair is slightly more tedious to use. But in order to make the task
easier, you may if you want use the following convenience class,
provided in the sis-storage module:
https://svn.apache.org/repos/asf/sis/trunk/storage/sis-storage/src/main/java/org/apache/sis/internal/storage/ChannelDataInput.java
Basically, you can give the ByteBuffer / ReadableChannel pair to the
constructor and either:
* Invoke ensureBufferContains(int) for specifying how many bytes you
are going to need for the next sequence of ByteBuffer.get*() operations;
* or use the convenience read*() methods when the number of bytes
needed is not known in advance.
Regards,
Martin
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Will review in short order.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Monday, June 3, 2013 7:40 PM
To: dev <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Uploaded the patch here.
>
>https://reviews.apache.org/r/11615/
>
>Thanks!
>Travis
>
>
>
>On Mon, Jun 3, 2013 at 10:28 PM, Mattmann, Chris A (398J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Thanks, Travis, great!
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Travis L Pinney <tr...@gmail.com>
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Monday, June 3, 2013 5:22 PM
>> To: dev <de...@sis.apache.org>
>> Subject: Re: Material for potential volunteer
>>
>> >I went ahead and checked in code that uses order() instead of extending
>> >MappedByteBuffer. After going back over the code, I noticed there was
>>not
>> >too many places that needed to be switched.
>> >
>> >I am working on the patch now.
>> >
>> >I am assuming this needs to go in sis/trunk/storage/sis-shapefile.
>> >
>> >
>> >Thanks,
>> >Travis
>> >
>> >
>> >
>> >On Mon, Jun 3, 2013 at 7:12 PM, Travis L Pinney
>> ><tr...@gmail.com>wrote:
>> >
>> >> Hello all,
>> >>
>> >> Just an update of the progress I made.
>> >>
>> >> 1. Switched to org.apache.sis.storage.shapefile for java/tests
>> >> 2. Truncated the test shapefiles and checked them in and deleted the
>> >> original larger shapefiles.
>> >> 3. Switched to using NIO
>> >>
>> >> The code is here
>> >>
>> >> https://github.com/tlpinney/shapefile-api
>> >>
>> >> Currently I have it set up to MMAP the files. I am still using Apache
>> >> EndianUtils for reading Little Endian values. I am not sure of the
>> >> advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of
>>using
>> >> the EndianUtils, but I will extend the MappedByteBuffer so the api is
>> >> easier to use when switching between Little Endian and Big Endian.
>> >>Either
>> >> method of reading Little Endian values can be swapped out without
>> >>changing
>> >> the api in the future.
>> >>
>> >> Maybe something like?
>> >>
>> >> foo.getLEInt()
>> >> foo.getLELong()
>> >>
>> >> for dealing with Little Endian values?
>> >>
>> >>
>> >> Thanks,
>> >> Travis
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Jun 3, 2013 at 2:17 PM, Adam Estrada
>> >><es...@gmail.com>wrote:
>> >>
>> >>> Thanks Travis!
>> >>>
>> >>>
>> >>> On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney
>> >>><travis.pinney@gmail.com
>> >>> >wrote:
>> >>>
>> >>> > Thanks for the suggestions.
>> >>> >
>> >>> > I will update the packaging to be consistent with apache-sis and
>>use
>> >>> > ReviewBoard for the patch.
>> >>> >
>> >>> > For the tests, I will regenerate the shapefiles with a subset of
>>the
>> >>> data
>> >>> > to keep them at a reasonable size. This will allow testing for all
>> >>> types of
>> >>> > data (Polyline, Shapefile, and Point for now). I agree it is not
>> >>>optimal
>> >>> > for basic tests to depend on large files.
>> >>> >
>> >>> > Using java.nio will be better than using RandomAccessFile. I will
>> >>>look
>> >>> into
>> >>> > getting that switched over.
>> >>> >
>> >>> >
>> >>> > Thanks!
>> >>> > Travis
>> >>> >
>> >>> >
>> >>> > On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
>> >>> > martin.desruisseaux@geomatys.fr> wrote:
>> >>> >
>> >>> > > Hello Travis
>> >>> > >
>> >>> > > Thanks for this work! I had a quick look at the code on GitHub,
>> >>>and I
>> >>> > > would like to do the following suggestions:
>> >>> > >
>> >>> > > *
>> >>> > >
>> >>> > > About the test data, the ANC90Ply_4326.dbf files could
>>easily be
>> >>> > > committed on SVN since it is only 19 kb. However the other
>>test
>> >>> > > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4
>>and
>> >>> 3.1
>> >>> > > Mb big. We have not yet established a mechanism for such
>>large
>> >>>test
>> >>> > > files. We may need to setup some FTP server for large test
>> >>>files,
>> >>> > > and design the tests in such a way that those tests are
>> >>>optional.
>> >>> > > Maybe for now it would be better to commit only
>> >>> ANC90Ply_4326.dbf...
>> >>> > >
>> >>> > > *
>> >>> > >
>> >>> > > The ShapeFile class uses java.io.RandomAccessFile for reading
>> >>>data,
>> >>> > > followed by calls to org.apache.commons.io.**EndianUtils for
>> >>> > > converting bytes to double (or other primitive types) while
>> >>>taking
>> >>> > > endianness in account. Would it be possible to use
>> >>> > > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer
>> >>> instead?
>> >>> > > It would take care of the above for you, potentially much
>>more
>> >>> > > efficiently.
>> >>> > >
>> >>> > >
>> >>> > > Thanks again!
>> >>> > >
>> >>> > > Martin
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > Le 03/06/13 01:47, Travis L Pinney a écrit :
>> >>> > >
>> >>> > > Hello everyone,
>> >>> > >>
>> >>> > >> I started working on a very rough prototype that can read in
>> >>> Shapefiles.
>> >>> > >>
>> >>> > >> https://github.com/tlpinney/**shapefile-api<
>> >>> > https://github.com/tlpinney/shapefile-api>
>> >>> > >>
>> >>> > >> In order to write a patch to submit, where would this component
>> >>> reside
>> >>> > in
>> >>> > >> the Apache SIS project?
>> >>> > >>
>> >>> > >>
>> >>> > >> Thanks,
>> >>> > >> Travis
>> >>> > >>
>> >>> > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >>
>>
>>
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
Uploaded the patch here.
https://reviews.apache.org/r/11615/
Thanks!
Travis
On Mon, Jun 3, 2013 at 10:28 PM, Mattmann, Chris A (398J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Thanks, Travis, great!
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Travis L Pinney <tr...@gmail.com>
> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
> Date: Monday, June 3, 2013 5:22 PM
> To: dev <de...@sis.apache.org>
> Subject: Re: Material for potential volunteer
>
> >I went ahead and checked in code that uses order() instead of extending
> >MappedByteBuffer. After going back over the code, I noticed there was not
> >too many places that needed to be switched.
> >
> >I am working on the patch now.
> >
> >I am assuming this needs to go in sis/trunk/storage/sis-shapefile.
> >
> >
> >Thanks,
> >Travis
> >
> >
> >
> >On Mon, Jun 3, 2013 at 7:12 PM, Travis L Pinney
> ><tr...@gmail.com>wrote:
> >
> >> Hello all,
> >>
> >> Just an update of the progress I made.
> >>
> >> 1. Switched to org.apache.sis.storage.shapefile for java/tests
> >> 2. Truncated the test shapefiles and checked them in and deleted the
> >> original larger shapefiles.
> >> 3. Switched to using NIO
> >>
> >> The code is here
> >>
> >> https://github.com/tlpinney/shapefile-api
> >>
> >> Currently I have it set up to MMAP the files. I am still using Apache
> >> EndianUtils for reading Little Endian values. I am not sure of the
> >> advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of using
> >> the EndianUtils, but I will extend the MappedByteBuffer so the api is
> >> easier to use when switching between Little Endian and Big Endian.
> >>Either
> >> method of reading Little Endian values can be swapped out without
> >>changing
> >> the api in the future.
> >>
> >> Maybe something like?
> >>
> >> foo.getLEInt()
> >> foo.getLELong()
> >>
> >> for dealing with Little Endian values?
> >>
> >>
> >> Thanks,
> >> Travis
> >>
> >>
> >>
> >>
> >> On Mon, Jun 3, 2013 at 2:17 PM, Adam Estrada
> >><es...@gmail.com>wrote:
> >>
> >>> Thanks Travis!
> >>>
> >>>
> >>> On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney
> >>><travis.pinney@gmail.com
> >>> >wrote:
> >>>
> >>> > Thanks for the suggestions.
> >>> >
> >>> > I will update the packaging to be consistent with apache-sis and use
> >>> > ReviewBoard for the patch.
> >>> >
> >>> > For the tests, I will regenerate the shapefiles with a subset of the
> >>> data
> >>> > to keep them at a reasonable size. This will allow testing for all
> >>> types of
> >>> > data (Polyline, Shapefile, and Point for now). I agree it is not
> >>>optimal
> >>> > for basic tests to depend on large files.
> >>> >
> >>> > Using java.nio will be better than using RandomAccessFile. I will
> >>>look
> >>> into
> >>> > getting that switched over.
> >>> >
> >>> >
> >>> > Thanks!
> >>> > Travis
> >>> >
> >>> >
> >>> > On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
> >>> > martin.desruisseaux@geomatys.fr> wrote:
> >>> >
> >>> > > Hello Travis
> >>> > >
> >>> > > Thanks for this work! I had a quick look at the code on GitHub,
> >>>and I
> >>> > > would like to do the following suggestions:
> >>> > >
> >>> > > *
> >>> > >
> >>> > > About the test data, the ANC90Ply_4326.dbf files could easily be
> >>> > > committed on SVN since it is only 19 kb. However the other test
> >>> > > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and
> >>> 3.1
> >>> > > Mb big. We have not yet established a mechanism for such large
> >>>test
> >>> > > files. We may need to setup some FTP server for large test
> >>>files,
> >>> > > and design the tests in such a way that those tests are
> >>>optional.
> >>> > > Maybe for now it would be better to commit only
> >>> ANC90Ply_4326.dbf...
> >>> > >
> >>> > > *
> >>> > >
> >>> > > The ShapeFile class uses java.io.RandomAccessFile for reading
> >>>data,
> >>> > > followed by calls to org.apache.commons.io.**EndianUtils for
> >>> > > converting bytes to double (or other primitive types) while
> >>>taking
> >>> > > endianness in account. Would it be possible to use
> >>> > > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer
> >>> instead?
> >>> > > It would take care of the above for you, potentially much more
> >>> > > efficiently.
> >>> > >
> >>> > >
> >>> > > Thanks again!
> >>> > >
> >>> > > Martin
> >>> > >
> >>> > >
> >>> > >
> >>> > > Le 03/06/13 01:47, Travis L Pinney a écrit :
> >>> > >
> >>> > > Hello everyone,
> >>> > >>
> >>> > >> I started working on a very rough prototype that can read in
> >>> Shapefiles.
> >>> > >>
> >>> > >> https://github.com/tlpinney/**shapefile-api<
> >>> > https://github.com/tlpinney/shapefile-api>
> >>> > >>
> >>> > >> In order to write a patch to submit, where would this component
> >>> reside
> >>> > in
> >>> > >> the Apache SIS project?
> >>> > >>
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Travis
> >>> > >>
> >>> > >
> >>> > >
> >>> >
> >>>
> >>
> >>
>
>
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks, Travis, great!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Monday, June 3, 2013 5:22 PM
To: dev <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>I went ahead and checked in code that uses order() instead of extending
>MappedByteBuffer. After going back over the code, I noticed there was not
>too many places that needed to be switched.
>
>I am working on the patch now.
>
>I am assuming this needs to go in sis/trunk/storage/sis-shapefile.
>
>
>Thanks,
>Travis
>
>
>
>On Mon, Jun 3, 2013 at 7:12 PM, Travis L Pinney
><tr...@gmail.com>wrote:
>
>> Hello all,
>>
>> Just an update of the progress I made.
>>
>> 1. Switched to org.apache.sis.storage.shapefile for java/tests
>> 2. Truncated the test shapefiles and checked them in and deleted the
>> original larger shapefiles.
>> 3. Switched to using NIO
>>
>> The code is here
>>
>> https://github.com/tlpinney/shapefile-api
>>
>> Currently I have it set up to MMAP the files. I am still using Apache
>> EndianUtils for reading Little Endian values. I am not sure of the
>> advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of using
>> the EndianUtils, but I will extend the MappedByteBuffer so the api is
>> easier to use when switching between Little Endian and Big Endian.
>>Either
>> method of reading Little Endian values can be swapped out without
>>changing
>> the api in the future.
>>
>> Maybe something like?
>>
>> foo.getLEInt()
>> foo.getLELong()
>>
>> for dealing with Little Endian values?
>>
>>
>> Thanks,
>> Travis
>>
>>
>>
>>
>> On Mon, Jun 3, 2013 at 2:17 PM, Adam Estrada
>><es...@gmail.com>wrote:
>>
>>> Thanks Travis!
>>>
>>>
>>> On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney
>>><travis.pinney@gmail.com
>>> >wrote:
>>>
>>> > Thanks for the suggestions.
>>> >
>>> > I will update the packaging to be consistent with apache-sis and use
>>> > ReviewBoard for the patch.
>>> >
>>> > For the tests, I will regenerate the shapefiles with a subset of the
>>> data
>>> > to keep them at a reasonable size. This will allow testing for all
>>> types of
>>> > data (Polyline, Shapefile, and Point for now). I agree it is not
>>>optimal
>>> > for basic tests to depend on large files.
>>> >
>>> > Using java.nio will be better than using RandomAccessFile. I will
>>>look
>>> into
>>> > getting that switched over.
>>> >
>>> >
>>> > Thanks!
>>> > Travis
>>> >
>>> >
>>> > On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
>>> > martin.desruisseaux@geomatys.fr> wrote:
>>> >
>>> > > Hello Travis
>>> > >
>>> > > Thanks for this work! I had a quick look at the code on GitHub,
>>>and I
>>> > > would like to do the following suggestions:
>>> > >
>>> > > *
>>> > >
>>> > > About the test data, the ANC90Ply_4326.dbf files could easily be
>>> > > committed on SVN since it is only 19 kb. However the other test
>>> > > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and
>>> 3.1
>>> > > Mb big. We have not yet established a mechanism for such large
>>>test
>>> > > files. We may need to setup some FTP server for large test
>>>files,
>>> > > and design the tests in such a way that those tests are
>>>optional.
>>> > > Maybe for now it would be better to commit only
>>> ANC90Ply_4326.dbf...
>>> > >
>>> > > *
>>> > >
>>> > > The ShapeFile class uses java.io.RandomAccessFile for reading
>>>data,
>>> > > followed by calls to org.apache.commons.io.**EndianUtils for
>>> > > converting bytes to double (or other primitive types) while
>>>taking
>>> > > endianness in account. Would it be possible to use
>>> > > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer
>>> instead?
>>> > > It would take care of the above for you, potentially much more
>>> > > efficiently.
>>> > >
>>> > >
>>> > > Thanks again!
>>> > >
>>> > > Martin
>>> > >
>>> > >
>>> > >
>>> > > Le 03/06/13 01:47, Travis L Pinney a écrit :
>>> > >
>>> > > Hello everyone,
>>> > >>
>>> > >> I started working on a very rough prototype that can read in
>>> Shapefiles.
>>> > >>
>>> > >> https://github.com/tlpinney/**shapefile-api<
>>> > https://github.com/tlpinney/shapefile-api>
>>> > >>
>>> > >> In order to write a patch to submit, where would this component
>>> reside
>>> > in
>>> > >> the Apache SIS project?
>>> > >>
>>> > >>
>>> > >> Thanks,
>>> > >> Travis
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
I went ahead and checked in code that uses order() instead of extending
MappedByteBuffer. After going back over the code, I noticed there was not
too many places that needed to be switched.
I am working on the patch now.
I am assuming this needs to go in sis/trunk/storage/sis-shapefile.
Thanks,
Travis
On Mon, Jun 3, 2013 at 7:12 PM, Travis L Pinney <tr...@gmail.com>wrote:
> Hello all,
>
> Just an update of the progress I made.
>
> 1. Switched to org.apache.sis.storage.shapefile for java/tests
> 2. Truncated the test shapefiles and checked them in and deleted the
> original larger shapefiles.
> 3. Switched to using NIO
>
> The code is here
>
> https://github.com/tlpinney/shapefile-api
>
> Currently I have it set up to MMAP the files. I am still using Apache
> EndianUtils for reading Little Endian values. I am not sure of the
> advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of using
> the EndianUtils, but I will extend the MappedByteBuffer so the api is
> easier to use when switching between Little Endian and Big Endian. Either
> method of reading Little Endian values can be swapped out without changing
> the api in the future.
>
> Maybe something like?
>
> foo.getLEInt()
> foo.getLELong()
>
> for dealing with Little Endian values?
>
>
> Thanks,
> Travis
>
>
>
>
> On Mon, Jun 3, 2013 at 2:17 PM, Adam Estrada <es...@gmail.com>wrote:
>
>> Thanks Travis!
>>
>>
>> On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney <travis.pinney@gmail.com
>> >wrote:
>>
>> > Thanks for the suggestions.
>> >
>> > I will update the packaging to be consistent with apache-sis and use
>> > ReviewBoard for the patch.
>> >
>> > For the tests, I will regenerate the shapefiles with a subset of the
>> data
>> > to keep them at a reasonable size. This will allow testing for all
>> types of
>> > data (Polyline, Shapefile, and Point for now). I agree it is not optimal
>> > for basic tests to depend on large files.
>> >
>> > Using java.nio will be better than using RandomAccessFile. I will look
>> into
>> > getting that switched over.
>> >
>> >
>> > Thanks!
>> > Travis
>> >
>> >
>> > On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
>> > martin.desruisseaux@geomatys.fr> wrote:
>> >
>> > > Hello Travis
>> > >
>> > > Thanks for this work! I had a quick look at the code on GitHub, and I
>> > > would like to do the following suggestions:
>> > >
>> > > *
>> > >
>> > > About the test data, the ANC90Ply_4326.dbf files could easily be
>> > > committed on SVN since it is only 19 kb. However the other test
>> > > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and
>> 3.1
>> > > Mb big. We have not yet established a mechanism for such large test
>> > > files. We may need to setup some FTP server for large test files,
>> > > and design the tests in such a way that those tests are optional.
>> > > Maybe for now it would be better to commit only
>> ANC90Ply_4326.dbf...
>> > >
>> > > *
>> > >
>> > > The ShapeFile class uses java.io.RandomAccessFile for reading data,
>> > > followed by calls to org.apache.commons.io.**EndianUtils for
>> > > converting bytes to double (or other primitive types) while taking
>> > > endianness in account. Would it be possible to use
>> > > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer
>> instead?
>> > > It would take care of the above for you, potentially much more
>> > > efficiently.
>> > >
>> > >
>> > > Thanks again!
>> > >
>> > > Martin
>> > >
>> > >
>> > >
>> > > Le 03/06/13 01:47, Travis L Pinney a écrit :
>> > >
>> > > Hello everyone,
>> > >>
>> > >> I started working on a very rough prototype that can read in
>> Shapefiles.
>> > >>
>> > >> https://github.com/tlpinney/**shapefile-api<
>> > https://github.com/tlpinney/shapefile-api>
>> > >>
>> > >> In order to write a patch to submit, where would this component
>> reside
>> > in
>> > >> the Apache SIS project?
>> > >>
>> > >>
>> > >> Thanks,
>> > >> Travis
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: Material for potential volunteer
Posted by Adam Estrada <es...@gmail.com>.
Thanks for working on this so diligently guys!
On Tue, Jun 4, 2013 at 9:06 AM, Travis L Pinney <tr...@gmail.com>wrote:
> Thanks for the suggestions. I will submit another patch.
>
> Cheers,
> Travis
>
>
>
> On Tue, Jun 4, 2013 at 5:11 AM, Martin Desruisseaux <
> martin.desruisseaux@geomatys.fr> wrote:
>
> > Le 04/06/13 01:12, Travis L Pinney a écrit :
> >
> >> Currently I have it set up to MMAP the files. I am still using Apache
> >> EndianUtils for reading Little Endian values. I am not sure of the
> >> advantage of calling foo.order(ByteOrder.LITTLE_**ENDIAN) instead of
> >> using
> >> the EndianUtils, (...snip...)
> >>
> >
> > I noticed that you switched to
> ByteBuffer.order(ByteOrder.**LITTLE_ENDIAN),
> > thanks. For the record, the advantages are:
> >
> > * No need for external libraries, since ByteBuffer is bundled in JDK
> 1.4+
> > * No need for additional API, since the same set of existing methods
> > work in both cases
> > * Potentially order of magnitude faster, since ByteBuffer.get*() with
> > Little Endian order maps directly to a single CPU instruction on
> > Intel processors, while I guess (but did not verified) that
> > EndianUtils performs arithmetic operations for achieving equivalent
> > result.
> >
> >
> > Martin
> >
> >
>
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
Thanks for the suggestions. I will submit another patch.
Cheers,
Travis
On Tue, Jun 4, 2013 at 5:11 AM, Martin Desruisseaux <
martin.desruisseaux@geomatys.fr> wrote:
> Le 04/06/13 01:12, Travis L Pinney a écrit :
>
>> Currently I have it set up to MMAP the files. I am still using Apache
>> EndianUtils for reading Little Endian values. I am not sure of the
>> advantage of calling foo.order(ByteOrder.LITTLE_**ENDIAN) instead of
>> using
>> the EndianUtils, (...snip...)
>>
>
> I noticed that you switched to ByteBuffer.order(ByteOrder.**LITTLE_ENDIAN),
> thanks. For the record, the advantages are:
>
> * No need for external libraries, since ByteBuffer is bundled in JDK 1.4+
> * No need for additional API, since the same set of existing methods
> work in both cases
> * Potentially order of magnitude faster, since ByteBuffer.get*() with
> Little Endian order maps directly to a single CPU instruction on
> Intel processors, while I guess (but did not verified) that
> EndianUtils performs arithmetic operations for achieving equivalent
> result.
>
>
> Martin
>
>
Re: Material for potential volunteer
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Le 04/06/13 01:12, Travis L Pinney a écrit :
> Currently I have it set up to MMAP the files. I am still using Apache
> EndianUtils for reading Little Endian values. I am not sure of the
> advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of using
> the EndianUtils, (...snip...)
I noticed that you switched to
ByteBuffer.order(ByteOrder.LITTLE_ENDIAN), thanks. For the record, the
advantages are:
* No need for external libraries, since ByteBuffer is bundled in JDK 1.4+
* No need for additional API, since the same set of existing methods
work in both cases
* Potentially order of magnitude faster, since ByteBuffer.get*() with
Little Endian order maps directly to a single CPU instruction on
Intel processors, while I guess (but did not verified) that
EndianUtils performs arithmetic operations for achieving equivalent
result.
Martin
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
Hello all,
Just an update of the progress I made.
1. Switched to org.apache.sis.storage.shapefile for java/tests
2. Truncated the test shapefiles and checked them in and deleted the
original larger shapefiles.
3. Switched to using NIO
The code is here
https://github.com/tlpinney/shapefile-api
Currently I have it set up to MMAP the files. I am still using Apache
EndianUtils for reading Little Endian values. I am not sure of the
advantage of calling foo.order(ByteOrder.LITTLE_ENDIAN) instead of using
the EndianUtils, but I will extend the MappedByteBuffer so the api is
easier to use when switching between Little Endian and Big Endian. Either
method of reading Little Endian values can be swapped out without changing
the api in the future.
Maybe something like?
foo.getLEInt()
foo.getLELong()
for dealing with Little Endian values?
Thanks,
Travis
On Mon, Jun 3, 2013 at 2:17 PM, Adam Estrada <es...@gmail.com> wrote:
> Thanks Travis!
>
>
> On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney <travis.pinney@gmail.com
> >wrote:
>
> > Thanks for the suggestions.
> >
> > I will update the packaging to be consistent with apache-sis and use
> > ReviewBoard for the patch.
> >
> > For the tests, I will regenerate the shapefiles with a subset of the data
> > to keep them at a reasonable size. This will allow testing for all types
> of
> > data (Polyline, Shapefile, and Point for now). I agree it is not optimal
> > for basic tests to depend on large files.
> >
> > Using java.nio will be better than using RandomAccessFile. I will look
> into
> > getting that switched over.
> >
> >
> > Thanks!
> > Travis
> >
> >
> > On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
> > martin.desruisseaux@geomatys.fr> wrote:
> >
> > > Hello Travis
> > >
> > > Thanks for this work! I had a quick look at the code on GitHub, and I
> > > would like to do the following suggestions:
> > >
> > > *
> > >
> > > About the test data, the ANC90Ply_4326.dbf files could easily be
> > > committed on SVN since it is only 19 kb. However the other test
> > > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and 3.1
> > > Mb big. We have not yet established a mechanism for such large test
> > > files. We may need to setup some FTP server for large test files,
> > > and design the tests in such a way that those tests are optional.
> > > Maybe for now it would be better to commit only ANC90Ply_4326.dbf...
> > >
> > > *
> > >
> > > The ShapeFile class uses java.io.RandomAccessFile for reading data,
> > > followed by calls to org.apache.commons.io.**EndianUtils for
> > > converting bytes to double (or other primitive types) while taking
> > > endianness in account. Would it be possible to use
> > > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer
> instead?
> > > It would take care of the above for you, potentially much more
> > > efficiently.
> > >
> > >
> > > Thanks again!
> > >
> > > Martin
> > >
> > >
> > >
> > > Le 03/06/13 01:47, Travis L Pinney a écrit :
> > >
> > > Hello everyone,
> > >>
> > >> I started working on a very rough prototype that can read in
> Shapefiles.
> > >>
> > >> https://github.com/tlpinney/**shapefile-api<
> > https://github.com/tlpinney/shapefile-api>
> > >>
> > >> In order to write a patch to submit, where would this component reside
> > in
> > >> the Apache SIS project?
> > >>
> > >>
> > >> Thanks,
> > >> Travis
> > >>
> > >
> > >
> >
>
Re: Material for potential volunteer
Posted by Adam Estrada <es...@gmail.com>.
Thanks Travis!
On Mon, Jun 3, 2013 at 10:04 AM, Travis L Pinney <tr...@gmail.com>wrote:
> Thanks for the suggestions.
>
> I will update the packaging to be consistent with apache-sis and use
> ReviewBoard for the patch.
>
> For the tests, I will regenerate the shapefiles with a subset of the data
> to keep them at a reasonable size. This will allow testing for all types of
> data (Polyline, Shapefile, and Point for now). I agree it is not optimal
> for basic tests to depend on large files.
>
> Using java.nio will be better than using RandomAccessFile. I will look into
> getting that switched over.
>
>
> Thanks!
> Travis
>
>
> On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
> martin.desruisseaux@geomatys.fr> wrote:
>
> > Hello Travis
> >
> > Thanks for this work! I had a quick look at the code on GitHub, and I
> > would like to do the following suggestions:
> >
> > *
> >
> > About the test data, the ANC90Ply_4326.dbf files could easily be
> > committed on SVN since it is only 19 kb. However the other test
> > files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and 3.1
> > Mb big. We have not yet established a mechanism for such large test
> > files. We may need to setup some FTP server for large test files,
> > and design the tests in such a way that those tests are optional.
> > Maybe for now it would be better to commit only ANC90Ply_4326.dbf...
> >
> > *
> >
> > The ShapeFile class uses java.io.RandomAccessFile for reading data,
> > followed by calls to org.apache.commons.io.**EndianUtils for
> > converting bytes to double (or other primitive types) while taking
> > endianness in account. Would it be possible to use
> > java.nio.channels.**ReadableChannel with java.nio.ByteBuffer instead?
> > It would take care of the above for you, potentially much more
> > efficiently.
> >
> >
> > Thanks again!
> >
> > Martin
> >
> >
> >
> > Le 03/06/13 01:47, Travis L Pinney a écrit :
> >
> > Hello everyone,
> >>
> >> I started working on a very rough prototype that can read in Shapefiles.
> >>
> >> https://github.com/tlpinney/**shapefile-api<
> https://github.com/tlpinney/shapefile-api>
> >>
> >> In order to write a patch to submit, where would this component reside
> in
> >> the Apache SIS project?
> >>
> >>
> >> Thanks,
> >> Travis
> >>
> >
> >
>
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Thanks Travis you rock!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Monday, June 3, 2013 7:04 AM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Thanks for the suggestions.
>
>I will update the packaging to be consistent with apache-sis and use
>ReviewBoard for the patch.
>
>For the tests, I will regenerate the shapefiles with a subset of the data
>to keep them at a reasonable size. This will allow testing for all types
>of
>data (Polyline, Shapefile, and Point for now). I agree it is not optimal
>for basic tests to depend on large files.
>
>Using java.nio will be better than using RandomAccessFile. I will look
>into
>getting that switched over.
>
>
>Thanks!
>Travis
>
>
>On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
>martin.desruisseaux@geomatys.fr> wrote:
>
>> Hello Travis
>>
>> Thanks for this work! I had a quick look at the code on GitHub, and I
>> would like to do the following suggestions:
>>
>> *
>>
>> About the test data, the ANC90Ply_4326.dbf files could easily be
>> committed on SVN since it is only 19 kb. However the other test
>> files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and 3.1
>> Mb big. We have not yet established a mechanism for such large test
>> files. We may need to setup some FTP server for large test files,
>> and design the tests in such a way that those tests are optional.
>> Maybe for now it would be better to commit only ANC90Ply_4326.dbf...
>>
>> *
>>
>> The ShapeFile class uses java.io.RandomAccessFile for reading data,
>> followed by calls to org.apache.commons.io.**EndianUtils for
>> converting bytes to double (or other primitive types) while taking
>> endianness in account. Would it be possible to use
>> java.nio.channels.**ReadableChannel with java.nio.ByteBuffer instead?
>> It would take care of the above for you, potentially much more
>> efficiently.
>>
>>
>> Thanks again!
>>
>> Martin
>>
>>
>>
>> Le 03/06/13 01:47, Travis L Pinney a écrit :
>>
>> Hello everyone,
>>>
>>> I started working on a very rough prototype that can read in
>>>Shapefiles.
>>>
>>>
>>>https://github.com/tlpinney/**shapefile-api<https://github.com/tlpinney/
>>>shapefile-api>
>>>
>>> In order to write a patch to submit, where would this component reside
>>>in
>>> the Apache SIS project?
>>>
>>>
>>> Thanks,
>>> Travis
>>>
>>
>>
Re: Material for potential volunteer
Posted by Travis L Pinney <tr...@gmail.com>.
Thanks for the suggestions.
I will update the packaging to be consistent with apache-sis and use
ReviewBoard for the patch.
For the tests, I will regenerate the shapefiles with a subset of the data
to keep them at a reasonable size. This will allow testing for all types of
data (Polyline, Shapefile, and Point for now). I agree it is not optimal
for basic tests to depend on large files.
Using java.nio will be better than using RandomAccessFile. I will look into
getting that switched over.
Thanks!
Travis
On Mon, Jun 3, 2013 at 4:37 AM, Martin Desruisseaux <
martin.desruisseaux@geomatys.fr> wrote:
> Hello Travis
>
> Thanks for this work! I had a quick look at the code on GitHub, and I
> would like to do the following suggestions:
>
> *
>
> About the test data, the ANC90Ply_4326.dbf files could easily be
> committed on SVN since it is only 19 kb. However the other test
> files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and 3.1
> Mb big. We have not yet established a mechanism for such large test
> files. We may need to setup some FTP server for large test files,
> and design the tests in such a way that those tests are optional.
> Maybe for now it would be better to commit only ANC90Ply_4326.dbf...
>
> *
>
> The ShapeFile class uses java.io.RandomAccessFile for reading data,
> followed by calls to org.apache.commons.io.**EndianUtils for
> converting bytes to double (or other primitive types) while taking
> endianness in account. Would it be possible to use
> java.nio.channels.**ReadableChannel with java.nio.ByteBuffer instead?
> It would take care of the above for you, potentially much more
> efficiently.
>
>
> Thanks again!
>
> Martin
>
>
>
> Le 03/06/13 01:47, Travis L Pinney a écrit :
>
> Hello everyone,
>>
>> I started working on a very rough prototype that can read in Shapefiles.
>>
>> https://github.com/tlpinney/**shapefile-api<https://github.com/tlpinney/shapefile-api>
>>
>> In order to write a patch to submit, where would this component reside in
>> the Apache SIS project?
>>
>>
>> Thanks,
>> Travis
>>
>
>
Re: Material for potential volunteer
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Hello Travis
Thanks for this work! I had a quick look at the code on GitHub, and I
would like to do the following suggestions:
*
About the test data, the ANC90Ply_4326.dbf files could easily be
committed on SVN since it is only 19 kb. However the other test
files (SignedBikeRoute_4326 and ABRALicenseePt_4326) are 2.4 and 3.1
Mb big. We have not yet established a mechanism for such large test
files. We may need to setup some FTP server for large test files,
and design the tests in such a way that those tests are optional.
Maybe for now it would be better to commit only ANC90Ply_4326.dbf...
*
The ShapeFile class uses java.io.RandomAccessFile for reading data,
followed by calls to org.apache.commons.io.EndianUtils for
converting bytes to double (or other primitive types) while taking
endianness in account. Would it be possible to use
java.nio.channels.ReadableChannel with java.nio.ByteBuffer instead?
It would take care of the above for you, potentially much more
efficiently.
Thanks again!
Martin
Le 03/06/13 01:47, Travis L Pinney a écrit :
> Hello everyone,
>
> I started working on a very rough prototype that can read in Shapefiles.
>
> https://github.com/tlpinney/shapefile-api
>
> In order to write a patch to submit, where would this component reside in
> the Apache SIS project?
>
>
> Thanks,
> Travis
Re: Material for potential volunteer
Posted by "Mattmann, Chris A (398J)" <ch...@jpl.nasa.gov>.
Hey Travis,
Awesome!
Steps:
1. Probably should live in
http://svn.apache.org/repos/asf/sis/trunk/storage/sis-shapefile/
2. Need to update packaging to match org.apache.sis.storage.shapefile in
your java/tests
3. You can use ReviewBoard if you want peeps to review the patch:
http://reviews.apache.org/
4. Please attach patch to SIS-100 and link to Review Board in #3
Looking forward to working with you!!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Travis L Pinney <tr...@gmail.com>
Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
Date: Sunday, June 2, 2013 4:47 PM
To: "dev@sis.apache.org" <de...@sis.apache.org>
Subject: Re: Material for potential volunteer
>Hello everyone,
>
>I started working on a very rough prototype that can read in Shapefiles.
>
>https://github.com/tlpinney/shapefile-api
>
>In order to write a patch to submit, where would this component reside in
>the Apache SIS project?
>
>
>Thanks,
>Travis
>
>
>
>
>On Wed, May 8, 2013 at 3:11 AM, Mattmann, Chris A (398J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Sounds like a great opportunity for someone to step up here :)
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Martin Desruisseaux <ma...@geomatys.fr>
>> Organization: Geomatys
>> Reply-To: "dev@sis.apache.org" <de...@sis.apache.org>
>> Date: Tuesday, May 7, 2013 8:00 AM
>> To: Apache SIS <de...@sis.apache.org>
>> Subject: Material for potential volunteer
>>
>> >Hello all
>> >
>> >We filled an "Implement Shapefile data store" JIRA task as an idea for
>> >potential volunteer. Actually it may be a bit early for implementing
>> >such data store since we don't have yet any DataStore interface, no
>> >Feature implementation, no Geometry and no CoordinateReferenceSystem.
>> >Nevertheless a draft based on java.util.Map may be possible if some
>> >volunteer wishes to put his hands in.
>> >
>> >https://issues.apache.org/jira/browse/SIS-100
>> >
>> >We filled this JIRA task because we can not port the Shapefile
>>datastore
>> >from Geotk for licensing reason (we can port many other datastores, but
>> >not that one). So ideally, it would be better to have someone else from
>> >the community to write at least a first draft. Johann Sorel, myself or
>> >others could continue after that point, building on the first draft.
>> >
>> >No rush however. If it takes some time before such work begins, it may
>> >leaves us the time to provide some of the missing bases.
>> >
>> > Martin
>> >
>>
>>