You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Kirk Lund <kl...@apache.org> on 2016/11/02 20:16:33 UTC
Apache policy on files types allowed in source code
I'd like to read up on what's allowed and disallowed regarding file types
in Apache source code. Does anyone have any links to get me started?
Thanks,
Kirk
Re: Apache policy on files types allowed in source code
Posted by Dan Smith <ds...@pivotal.io>.
I don't know about apache policy, but I think anything that can't be easily
modified is probably a bad thing to have in the source code. If there is a
binary jar, but there's no way to update the jar to modify the test that
seems like a problem.
Since we have a framework to run tests with old versions and new versions
of geode, I don't see why there should be any reason to check in artifacts
generated by old versions of geode either.
Can you describe the specific tests? I saw Anthony's downvote on the
release, but it's not clear what tests use those artifacts or what they are
for.
-Dan
On Mon, Feb 6, 2017 at 9:28 AM, Kirk Lund <kl...@apache.org> wrote:
> [bump]
>
> I want to see a definitive reference on what's allowed to be in the src of
> an Apache project before we take actions to change any tests that use
> binary resources.
>
> We have unit tests and regression tests that are using binaries (including
> stat files, diskstore files, class files). Jumping through hoops to execute
> a test that requires an old binary that was generated by old geode src
> prior to a bug fix (stats, diskstores, etc) or that needs to compile
> against old geode src (backwards compatibility of user callbacks) seems
> like a huge waste of time and energy.
>
> Please provide more info on list if anyone has issues with what's in geode
> src code.
>
> Thanks,
> Kirk
>
>
> On Wed, Nov 2, 2016 at 1:46 PM, Kirk Lund <kl...@apache.org> wrote:
>
> > I do have a particular case in mind involving stats files that geode
> > creates and reads. The StatArchiveWriter.java class writes out statistic
> > archive files for geode, while StatArchiveReader.java reads them in.
> >
> > There have been a few bugs causing the stats files to be corrupted in
> some
> > way. My goal has been to fix the StatArchiveWriter to prevent any
> > corruption, but also fix the StatArchiveReader to self-correct where
> > possible or provide better error handling if self-correction is not
> > possible.
> >
> > The result is that we have a couple regression tests under
> > geode-core/src/test/java with corresponding corrupted statistic archive
> > files under geode-core/src/test/resources.
> >
> > Example: I currently have a review posted for GEODE-2013 https://reviews
> .
> > apache.org/r/53402 with these files:
> >
> > geode-core/src/main/java/org/apache/geode/internal/
> > statistics/StatArchiveReader.java
> > geode-core/src/test/java/org/apache/geode/internal/statistics/
> > StatArchiveWithMissingResourceTypeRegressionTest.java
> > geode-core/src/test/resources/org/apache/geode/internal/statistics/
> > StatArchiveWithMissingResourceTypeRegressionTest.gfs
> >
> > The above file StatArchiveWithMissingResourceTypeRegressionTest.gfs is
> > small (253 bytes on Mac). After fixing the source of the problem in
> > StatArchiveWriter (GEODE-2012), GEODE-2013 involves providing better
> error
> > messaging to the user if they try to read an older stats file that was
> > written before GEODE-2012 was fixed.
> >
> > Thanks,
> > Kirk
> >
> > On Wed, Nov 2, 2016 at 1:19 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> > wrote:
> >
> >> On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
> >> > I'd like to read up on what's allowed and disallowed regarding file
> >> types
> >> > in Apache source code. Does anyone have any links to get me started?
> >>
> >> The only distinction that ASF makes is binaries vs. source files. The
> >> source
> >> code that ASF releases has to be comprised of files that can be called
> >> source (under a reasonable definition of source). The only requirement
> >> for source files is that they MUST be usable without leveraging
> commercial
> >> tools. Other than that -- anything goes.
> >>
> >> Are you asking with a particular example in mind or just asking a
> >> hypothetical?
> >>
> >> Thanks,
> >> Roman.
> >>
> >
> >
>
Re: Apache policy on files types allowed in source code
Posted by Dave Barnes <db...@pivotal.io>.
The docs directories contain binary files in various graphic formats: svg,
png, gif, jpg and the like.
I haven't made a practice of checking in the 'sources' for these, because
they're nearly always created with proprietary tools, such as Visio or
OmniGraffle. The workflow for graphics is different from that of compiled
code, as contributors in the community can update these graphics by
starting with the binaries and using their own tools, then push their
binary output to the repo. No original source required, even though the
file in the repo is in some sense a binary 'artifact'.
On Mon, Feb 6, 2017 at 10:33 AM, Anthony Baker <ab...@pivotal.io> wrote:
> Well, here’s a rule of thumb: if the file was generated by a compiler
> it’s probably not source :-)
>
> Binary files such as images, diskstores, stats, etc should ok. There are
> lots of discussions on this topic, see [1] for example. Take a look at our
> current list of rat excludes in ‘gradle/rat.gradle’.
>
> Anthony
>
> [1] http://mail-archives.apache.org/mod_mbox/www-legal-
> discuss/201701.mbox/%3cCAOqetn_SQQV8z3tD+XtGiBNBXGEmogUbju-
> 2aFAsOQBSROWK2A@mail.gmail.com%3e
>
>
> > On Feb 6, 2017, at 9:28 AM, Kirk Lund <kl...@apache.org> wrote:
> >
> > [bump]
> >
> > I want to see a definitive reference on what's allowed to be in the src
> of
> > an Apache project before we take actions to change any tests that use
> > binary resources.
> >
> > We have unit tests and regression tests that are using binaries
> (including
> > stat files, diskstore files, class files). Jumping through hoops to
> execute
> > a test that requires an old binary that was generated by old geode src
> > prior to a bug fix (stats, diskstores, etc) or that needs to compile
> > against old geode src (backwards compatibility of user callbacks) seems
> > like a huge waste of time and energy.
> >
> > Please provide more info on list if anyone has issues with what's in
> geode
> > src code.
> >
> > Thanks,
> > Kirk
> >
> >
> > On Wed, Nov 2, 2016 at 1:46 PM, Kirk Lund <kl...@apache.org> wrote:
> >
> >> I do have a particular case in mind involving stats files that geode
> >> creates and reads. The StatArchiveWriter.java class writes out statistic
> >> archive files for geode, while StatArchiveReader.java reads them in.
> >>
> >> There have been a few bugs causing the stats files to be corrupted in
> some
> >> way. My goal has been to fix the StatArchiveWriter to prevent any
> >> corruption, but also fix the StatArchiveReader to self-correct where
> >> possible or provide better error handling if self-correction is not
> >> possible.
> >>
> >> The result is that we have a couple regression tests under
> >> geode-core/src/test/java with corresponding corrupted statistic archive
> >> files under geode-core/src/test/resources.
> >>
> >> Example: I currently have a review posted for GEODE-2013
> https://reviews.
> >> apache.org/r/53402 with these files:
> >>
> >> geode-core/src/main/java/org/apache/geode/internal/
> >> statistics/StatArchiveReader.java
> >> geode-core/src/test/java/org/apache/geode/internal/statistics/
> >> StatArchiveWithMissingResourceTypeRegressionTest.java
> >> geode-core/src/test/resources/org/apache/geode/internal/statistics/
> >> StatArchiveWithMissingResourceTypeRegressionTest.gfs
> >>
> >> The above file StatArchiveWithMissingResourceTypeRegressionTest.gfs is
> >> small (253 bytes on Mac). After fixing the source of the problem in
> >> StatArchiveWriter (GEODE-2012), GEODE-2013 involves providing better
> error
> >> messaging to the user if they try to read an older stats file that was
> >> written before GEODE-2012 was fixed.
> >>
> >> Thanks,
> >> Kirk
> >>
> >> On Wed, Nov 2, 2016 at 1:19 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> >> wrote:
> >>
> >>> On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
> >>>> I'd like to read up on what's allowed and disallowed regarding file
> >>> types
> >>>> in Apache source code. Does anyone have any links to get me started?
> >>>
> >>> The only distinction that ASF makes is binaries vs. source files. The
> >>> source
> >>> code that ASF releases has to be comprised of files that can be called
> >>> source (under a reasonable definition of source). The only requirement
> >>> for source files is that they MUST be usable without leveraging
> commercial
> >>> tools. Other than that -- anything goes.
> >>>
> >>> Are you asking with a particular example in mind or just asking a
> >>> hypothetical?
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
>
>
Re: Apache policy on files types allowed in source code
Posted by Anthony Baker <ab...@pivotal.io>.
Well, here’s a rule of thumb: if the file was generated by a compiler it’s probably not source :-)
Binary files such as images, diskstores, stats, etc should ok. There are lots of discussions on this topic, see [1] for example. Take a look at our current list of rat excludes in ‘gradle/rat.gradle’.
Anthony
[1] http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201701.mbox/%3cCAOqetn_SQQV8z3tD+XtGiBNBXGEmogUbju-2aFAsOQBSROWK2A@mail.gmail.com%3e
> On Feb 6, 2017, at 9:28 AM, Kirk Lund <kl...@apache.org> wrote:
>
> [bump]
>
> I want to see a definitive reference on what's allowed to be in the src of
> an Apache project before we take actions to change any tests that use
> binary resources.
>
> We have unit tests and regression tests that are using binaries (including
> stat files, diskstore files, class files). Jumping through hoops to execute
> a test that requires an old binary that was generated by old geode src
> prior to a bug fix (stats, diskstores, etc) or that needs to compile
> against old geode src (backwards compatibility of user callbacks) seems
> like a huge waste of time and energy.
>
> Please provide more info on list if anyone has issues with what's in geode
> src code.
>
> Thanks,
> Kirk
>
>
> On Wed, Nov 2, 2016 at 1:46 PM, Kirk Lund <kl...@apache.org> wrote:
>
>> I do have a particular case in mind involving stats files that geode
>> creates and reads. The StatArchiveWriter.java class writes out statistic
>> archive files for geode, while StatArchiveReader.java reads them in.
>>
>> There have been a few bugs causing the stats files to be corrupted in some
>> way. My goal has been to fix the StatArchiveWriter to prevent any
>> corruption, but also fix the StatArchiveReader to self-correct where
>> possible or provide better error handling if self-correction is not
>> possible.
>>
>> The result is that we have a couple regression tests under
>> geode-core/src/test/java with corresponding corrupted statistic archive
>> files under geode-core/src/test/resources.
>>
>> Example: I currently have a review posted for GEODE-2013 https://reviews.
>> apache.org/r/53402 with these files:
>>
>> geode-core/src/main/java/org/apache/geode/internal/
>> statistics/StatArchiveReader.java
>> geode-core/src/test/java/org/apache/geode/internal/statistics/
>> StatArchiveWithMissingResourceTypeRegressionTest.java
>> geode-core/src/test/resources/org/apache/geode/internal/statistics/
>> StatArchiveWithMissingResourceTypeRegressionTest.gfs
>>
>> The above file StatArchiveWithMissingResourceTypeRegressionTest.gfs is
>> small (253 bytes on Mac). After fixing the source of the problem in
>> StatArchiveWriter (GEODE-2012), GEODE-2013 involves providing better error
>> messaging to the user if they try to read an older stats file that was
>> written before GEODE-2012 was fixed.
>>
>> Thanks,
>> Kirk
>>
>> On Wed, Nov 2, 2016 at 1:19 PM, Roman Shaposhnik <ro...@shaposhnik.org>
>> wrote:
>>
>>> On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
>>>> I'd like to read up on what's allowed and disallowed regarding file
>>> types
>>>> in Apache source code. Does anyone have any links to get me started?
>>>
>>> The only distinction that ASF makes is binaries vs. source files. The
>>> source
>>> code that ASF releases has to be comprised of files that can be called
>>> source (under a reasonable definition of source). The only requirement
>>> for source files is that they MUST be usable without leveraging commercial
>>> tools. Other than that -- anything goes.
>>>
>>> Are you asking with a particular example in mind or just asking a
>>> hypothetical?
>>>
>>> Thanks,
>>> Roman.
>>>
>>
>>
Re: Apache policy on files types allowed in source code
Posted by Kirk Lund <kl...@apache.org>.
[bump]
I want to see a definitive reference on what's allowed to be in the src of
an Apache project before we take actions to change any tests that use
binary resources.
We have unit tests and regression tests that are using binaries (including
stat files, diskstore files, class files). Jumping through hoops to execute
a test that requires an old binary that was generated by old geode src
prior to a bug fix (stats, diskstores, etc) or that needs to compile
against old geode src (backwards compatibility of user callbacks) seems
like a huge waste of time and energy.
Please provide more info on list if anyone has issues with what's in geode
src code.
Thanks,
Kirk
On Wed, Nov 2, 2016 at 1:46 PM, Kirk Lund <kl...@apache.org> wrote:
> I do have a particular case in mind involving stats files that geode
> creates and reads. The StatArchiveWriter.java class writes out statistic
> archive files for geode, while StatArchiveReader.java reads them in.
>
> There have been a few bugs causing the stats files to be corrupted in some
> way. My goal has been to fix the StatArchiveWriter to prevent any
> corruption, but also fix the StatArchiveReader to self-correct where
> possible or provide better error handling if self-correction is not
> possible.
>
> The result is that we have a couple regression tests under
> geode-core/src/test/java with corresponding corrupted statistic archive
> files under geode-core/src/test/resources.
>
> Example: I currently have a review posted for GEODE-2013 https://reviews.
> apache.org/r/53402 with these files:
>
> geode-core/src/main/java/org/apache/geode/internal/
> statistics/StatArchiveReader.java
> geode-core/src/test/java/org/apache/geode/internal/statistics/
> StatArchiveWithMissingResourceTypeRegressionTest.java
> geode-core/src/test/resources/org/apache/geode/internal/statistics/
> StatArchiveWithMissingResourceTypeRegressionTest.gfs
>
> The above file StatArchiveWithMissingResourceTypeRegressionTest.gfs is
> small (253 bytes on Mac). After fixing the source of the problem in
> StatArchiveWriter (GEODE-2012), GEODE-2013 involves providing better error
> messaging to the user if they try to read an older stats file that was
> written before GEODE-2012 was fixed.
>
> Thanks,
> Kirk
>
> On Wed, Nov 2, 2016 at 1:19 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
>> > I'd like to read up on what's allowed and disallowed regarding file
>> types
>> > in Apache source code. Does anyone have any links to get me started?
>>
>> The only distinction that ASF makes is binaries vs. source files. The
>> source
>> code that ASF releases has to be comprised of files that can be called
>> source (under a reasonable definition of source). The only requirement
>> for source files is that they MUST be usable without leveraging commercial
>> tools. Other than that -- anything goes.
>>
>> Are you asking with a particular example in mind or just asking a
>> hypothetical?
>>
>> Thanks,
>> Roman.
>>
>
>
Re: Apache policy on files types allowed in source code
Posted by Kirk Lund <kl...@apache.org>.
I do have a particular case in mind involving stats files that geode
creates and reads. The StatArchiveWriter.java class writes out statistic
archive files for geode, while StatArchiveReader.java reads them in.
There have been a few bugs causing the stats files to be corrupted in some
way. My goal has been to fix the StatArchiveWriter to prevent any
corruption, but also fix the StatArchiveReader to self-correct where
possible or provide better error handling if self-correction is not
possible.
The result is that we have a couple regression tests under
geode-core/src/test/java with corresponding corrupted statistic archive
files under geode-core/src/test/resources.
Example: I currently have a review posted for GEODE-2013
https://reviews.apache.org/r/53402 with these files:
geode-core/src/main/java/org/apache/geode/internal/statistics/StatArchiveReader.java
geode-core/src/test/java/org/apache/geode/internal/statistics/StatArchiveWithMissingResourceTypeRegressionTest.java
geode-core/src/test/resources/org/apache/geode/internal/statistics/StatArchiveWithMissingResourceTypeRegressionTest.gfs
The above file StatArchiveWithMissingResourceTypeRegressionTest.gfs is
small (253 bytes on Mac). After fixing the source of the problem in
StatArchiveWriter (GEODE-2012), GEODE-2013 involves providing better error
messaging to the user if they try to read an older stats file that was
written before GEODE-2012 was fixed.
Thanks,
Kirk
On Wed, Nov 2, 2016 at 1:19 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
> > I'd like to read up on what's allowed and disallowed regarding file types
> > in Apache source code. Does anyone have any links to get me started?
>
> The only distinction that ASF makes is binaries vs. source files. The
> source
> code that ASF releases has to be comprised of files that can be called
> source (under a reasonable definition of source). The only requirement
> for source files is that they MUST be usable without leveraging commercial
> tools. Other than that -- anything goes.
>
> Are you asking with a particular example in mind or just asking a
> hypothetical?
>
> Thanks,
> Roman.
>
Re: Apache policy on files types allowed in source code
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Nov 2, 2016 at 1:16 PM, Kirk Lund <kl...@apache.org> wrote:
> I'd like to read up on what's allowed and disallowed regarding file types
> in Apache source code. Does anyone have any links to get me started?
The only distinction that ASF makes is binaries vs. source files. The source
code that ASF releases has to be comprised of files that can be called
source (under a reasonable definition of source). The only requirement
for source files is that they MUST be usable without leveraging commercial
tools. Other than that -- anything goes.
Are you asking with a particular example in mind or just asking a hypothetical?
Thanks,
Roman.