Re: Questions about the Reference Collection dependencies. Was: Re: Release 3.0

[Peter <ji...@zeus.net.au>]

Bryan,

The only CI tests run on a regular basis are for qa-refactor, it is 
relatively simple to configure these tests to run the new branch, the 
jtreg tests have to be run manually however, jtreg is installed on 
Hudson, but incorrectly, it needs to be raised as an issue with the 
hudson admin.

QA Tests fail due to ports in use, or classdep related failures where 
some class files are missing from jar files, I ignore these test 
failures.  I do however, save tests that fail for other reasons.

The intent over the long term is to deprecate classdep and move to a 
maven build, Dennis has some scripts for this purpose included in this 
build, the qa test suite will also require modularisation, but should 
probably be broken out separately to simplify the entire process.

We have three test suites:

   1. The integration test suite.
   2. The jtreg test suite.
   3. The junit test suite.

Our timezones are 14 hours apart, I'm GMT+10, you appear to be GMT-4.

Regards,

Peter.

On 1/09/2015 3:16 AM, Bryan Thompson wrote:
> Is the branch that Dennis worked on in CI yet?  I am looking at [1] and I
> do not see it called out explicitly.  In fact, it looks like the full CI is
> only running against the trunk.  If that is true, can we put Dennis's
> branch into CI?  And it looks like CI has several pieces.  I assume they
> all need to be running?
>
> Thanks,
> Bryan
>
> [1] https://river.apache.org/ci.html
>
> ----
> Bryan Thompson
> Chief Scientist&  Founder
> SYSTAP, LLC
> 4501 Tower Road
> Greensboro, NC 27410
> bryan@systap.com
> http://blazegraph.com
> http://blog.bigdata.com<http://bigdata.com>
> http://mapgraph.io
>
> Blazegraph™<http://www.blazegraph.com/>  is our ultra high-performance
> graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
> APIs.  Blazegraph is now available with GPU acceleration using our disruptive
> technology to accelerate data-parallel graph analytics and graph query.
>
> CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
> for the sole use of the intended recipient(s) and are confidential or
> proprietary to SYSTAP. Any unauthorized review, use, disclosure,
> dissemination or copying of this email or its contents or attachments is
> prohibited. If you have received this communication in error, please notify
> the sender by reply email and permanently delete all copies of the email
> and its contents and attachments.
>
> On Mon, Aug 31, 2015 at 10:36 AM, Greg Trasuk<tr...@stratuscom.com>
> wrote:
>
>> Yes and no.  He put the sources in a jar file inside the deps-lib/rc-libs
>> folder in the qa-refactor branch.
>>
>> Branch is at:
>> https://svn.apache.org/repos/asf/river/jtsk/skunk/qa_refactor/trunk/<
>> https://svn.apache.org/repos/asf/river/jtsk/skunk/qa_refactor/trunk/>
>>
>> or Dennis’s namespace refactoring at
>>
>>
>> https://svn.apache.org/repos/asf/river/jtsk/skunk/qa-refactor-namespace/trunk/
>> <
>> https://svn.apache.org/repos/asf/river/jtsk/skunk/qa-refactor-namespace/trunk/
>>
>> Cheers,
>>
>> Greg Trasuk
>>
>>> On Aug 31, 2015, at 10:18 AM, Bryan Thompson<br...@systap.com>  wrote:
>>>
>>> I am pretty sure Peter also indicated that the code is in fact checked in
>>> to apache river right now - part of the qa branch as I recall.  Thus it
>> is
>>> already contributed by Peter.  Right?  I believe that all that is
>> required
>>> is a package rename into org.apache.river....
>>>
>>> Can someone please confirm the current branch and how to check it out?
>> Per
>>> [1], this would be the trunk.  I just would like to make sure that I am
>>> looking at the right branch for these discussions.
>>>
>>> svn checkout http://svn.apache.org/repos/asf/river/jtsk/trunk river
>>>
>>>
>>> Thanks,
>>> Bryan
>>>
>>> [1] https://river.apache.org/source-code.html
>>>
>>> ----
>>> Bryan Thompson
>>> Chief Scientist&  Founder
>>> SYSTAP, LLC
>>> 4501 Tower Road
>>> Greensboro, NC 27410
>>> bryan@systap.com
>>> http://blazegraph.com
>>> http://blog.bigdata.com<http://bigdata.com>
>>> http://mapgraph.io
>>>
>>> Blazegraph™<http://www.blazegraph.com/>  is our ultra high-performance
>>> graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
>>> APIs.  Blazegraph is now available with GPU acceleration using our
>> disruptive
>>> technology to accelerate data-parallel graph analytics and graph query.
>>>
>>> CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
>>> for the sole use of the intended recipient(s) and are confidential or
>>> proprietary to SYSTAP. Any unauthorized review, use, disclosure,
>>> dissemination or copying of this email or its contents or attachments is
>>> prohibited. If you have received this communication in error, please
>> notify
>>> the sender by reply email and permanently delete all copies of the email
>>> and its contents and attachments.
>>>
>>> On Mon, Aug 31, 2015 at 10:07 AM, Patricia Shanahan<pa...@acm.org>
>> wrote:
>>>> Although Peter has indicated approval, from a licensing point of view it
>>>> might be best if he were the one to check it in.
>>>>
>>>>
>>>> On 8/31/2015 6:11 AM, Bryan Thompson wrote:
>>>>
>>>>> Fine by me.  I am pretty sure Peter already indicated approval for
>> this.
>>>>> I
>>>>> was just offering to help remove a potential blocker for the release.
>>>>>
>>>>> +1 for publishing apple custard as a river artifact.
>>>>>
>>>>> Bryan
>>>>>
>>>>> ----
>>>>> Bryan Thompson
>>>>> Chief Scientist&  Founder
>>>>> SYSTAP, LLC
>>>>> 4501 Tower Road
>>>>> Greensboro, NC 27410
>>>>> bryan@systap.com
>>>>> http://blazegraph.com
>>>>> http://blog.bigdata.com<http://bigdata.com>
>>>>> http://mapgraph.io
>>>>>
>>>>> Blazegraph™<http://www.blazegraph.com/>  is our ultra high-performance
>>>>> graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
>>>>> APIs.  Blazegraph is now available with GPU acceleration using our
>>>>> disruptive
>>>>> technology to accelerate data-parallel graph analytics and graph query.
>>>>>
>>>>> CONFIDENTIALITY NOTICE:  This email and its contents and attachments
>> are
>>>>> for the sole use of the intended recipient(s) and are confidential or
>>>>> proprietary to SYSTAP. Any unauthorized review, use, disclosure,
>>>>> dissemination or copying of this email or its contents or attachments
>> is
>>>>> prohibited. If you have received this communication in error, please
>>>>> notify
>>>>> the sender by reply email and permanently delete all copies of the
>> email
>>>>> and its contents and attachments.
>>>>>
>>>>> On Mon, Aug 31, 2015 at 9:04 AM, Greg Trasuk<tr...@stratuscom.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>> If Peter is OK with it, it’s simpler for the project if the code gets
>>>>>> integrated into River.  At some point I saw an
>>>>>> ‘org.apache.river.collections’ package, which I thought was the
>>>>>> custard-apple thing.
>>>>>>
>>>>>> If there’s a strong need for the artifact to be published on its own,
>> it
>>>>>> could also be done through River.  We’re perfectly free to publish
>> more
>>>>>> than one artifact under the org.apache.river.* group id.  We would
>> need
>>>>>> to
>>>>>> create a subversion or git repository for it and then vote a release,
>> the
>>>>>> same as any other release.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Greg Trasuk
>>>>>>
>>>>>>> On Aug 31, 2015, at 3:37 AM, Peter<ji...@zeus.net.au>  wrote:
>>>>>>>
>>>>>>> I'd appreciate it Bryan,  use the version in the qa_suite, the
>> version
>>>>>> on Sourceforge is older.
>>>>>>
>>>>>>>
>>>>>>>
>> http://svn.apache.org/viewvc/river/jtsk/skunk/qa_refactor/trunk/dep-libs/rc-libs/
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Peter.
>>>>>>>
>>>>>>> On 30/08/2015 9:54 PM, Bryan Thompson wrote:
>>>>>>>
>>>>>>>> Peter,  would you be open to having someone else publish the
>> artifact
>>>>>>>> to
>>>>>>>> maven? We are pretty deep in the process of publishing out a
>> variety of
>>>>>>>> maven artifacts.  Brad  (cc) might be amenable to doing this to help
>>>>>>>>
>>>>>>> remove
>>>>>>> possible blockers from a 3.0 river release.
>>>>>>>> Just fyi, we are in US eastern so the time zone difference is pretty
>>>>>>>>
>>>>>>> large.
>>>>>>>> Bryan
>>>>>>>> On Aug 30, 2015 6:47 AM, "Peter"<ji...@zeus.net.au>   wrote:
>>>>>>>>
>>>>>>>> My uploading custard-apple to Maven Central will be dependant on me
>>>>>>>> creating new PGP Keys in a Unix environment, which I also have to
>>>>>>>>
>>>>>>> install,
>>>>>>> I do intend on getting this done in the near future, however I'm also
>>>>>>> quite
>>>>>>> busy, but given Rivers pace of devlopment, I'll probably get this
>> done
>>>>>>>> before 3.0 is released.
>>>>>>>>
>>>>>>>> I'm not against any options below, the project is free to choose.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Peter.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 13/08/2015 6:49 PM, Patricia Shanahan wrote:
>>>>>>>>
>>>>>>>> Good point. That seems to eliminate my option 1.
>>>>>>>>> On 8/13/2015 1:45 AM, Greg Trasuk wrote:
>>>>>>>>>
>>>>>>>>> If we have a dependency on a library that’s not in Maven Central,
>>>>>>>>>> then using River in a Maven-based project (for example, the River
>>>>>>>>>> Examples project) will effectively be impossible (it can be done
>> but
>>>>>>>>>> is a royal nuisance for downstream users).
>>>>>>>>>>
>>>>>>>>>> As such, I’d vote against any release that has a dependency on a
>>>>>>>>>> library that’s not in Maven Central.
>>>>>>>>>>
>>>>>>>>>> If Peter’s not able to put custard-apple into Maven Central, then
>> I
>>>>>>>>>> think we have no choice but to accept it as contributed code and
>> roll
>>>>>>>>>> it into River.
>>>>>>>>>>
>>>>>>>>>> A third option would be to accept it as a contributed code, but
>> treat
>>>>>>>>>> it as a separate deliverable and publish it on its own into Maven
>>>>>>>>>> Central.  The only caveat is that we would have to rename the
>>>>>>>>>> packages to ‘org.apache.river….’, since we can only publish under
>> the
>>>>>>>>>> ‘org.apache’ and ‘net.jini’ group ids.  Peter (or anyone here)
>> could
>>>>>>>>>> maintain it under River’s aegis, and other users could still get
>> it
>>>>>>>>>> out of Maven Central.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Greg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Aug 12, 2015, at 3:24 PM, Patricia Shanahan<pa...@acm.org>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Peter.
>>>>>>>>>>>
>>>>>>>>>>> It appears to me that we have three options for dealing with
>>>>>>>>>>> "custard apple":
>>>>>>>>>>>
>>>>>>>>>>> 1. Do not include it in the release, but include a download link
>> in
>>>>>>>>>>> the installation instructions.
>>>>>>>>>>>
>>>>>>>>>>> Pro: easy for us. Con: more work for users.
>>>>>>>>>>>
>>>>>>>>>>> 2. Include a selected source subset that River currently needs.
>>>>>>>>>>>
>>>>>>>>>>> Pro: minimize source code volume. Con: more complicated
>> maintenance
>>>>>>>>>>> - if Peter makes a change, we would need to check whether it
>>>>>>>>>>> involves the subset.
>>>>>>>>>>>
>>>>>>>>>>> 3. Include the whole library source.
>>>>>>>>>>>
>>>>>>>>>>> Pro: simpler maintenance - if Peter makes a change, we just copy
>> in
>>>>>>>>>>> the change. Con: Distributing source code that is not used by
>>>>>>>>>>> River.
>>>>>>>>>>>
>>>>>>>>>>> Any other options we should consider? Other pros or cons I have
>> not
>>>>>>>>>>> thought of? Opinions?
>>>>>>>>>>>
>>>>>>>>>>> On 8/12/2015 5:34 AM, Peter wrote:
>>>>>>>>>>>
>>>>>>>>>>> It's AL2 licensed, the source is in a jar file in dep-libs,
>>>>>>>>>>>> consider it already contributed.
>>>>>>>>>>>>
>>>>>>>>>>>> The library contains more code than you need, as it covers every
>>>>>>>>>>>> type of Java Collections interface, up to Java 7 (it will be
>>>>>>>>>>>> updated at some point to support Java 8&   9 collections
>>>>>>>>>>>> interfaces also).
>>>>>>>>>>>>
>>>>>>>>>>>> The code itself is scalable, it has a single JVM thread that
>>>>>>>>>>>> performs garbage collection of references, api calling threads
>>>>>>>>>>>> don't perform this task.  The references themselves are often
>>>>>>>>>>>> created and destroyed without being shared among other threads,
>> a
>>>>>>>>>>>> large number never leave CPU cache and safe publication is used
>>>>>>>>>>>> to avoid synchronization.  Overhead is low and its public API is
>>>>>>>>>>>> compact.
>>>>>>>>>>>>
>>>>>>>>>>>> For the most part use of soft references is advised against,
>>>>>>>>>>>> however there is a case for a cache that degrades gracefully
>>>>>>>>>>>> under load in the form of a Map where keys are timed and values
>>>>>>>>>>>> softly reachable (or vice versa), to avoid extreme cases where a
>>>>>>>>>>>> JVM is running out of memory, the jvm gc can clear entries that
>>>>>>>>>>>> aren't strongly reachable (even though timed keys haven't
>>>>>>>>>>>> expired) at its descretion potentially avoiding or at least
>>>>>>>>>>>> delaying an OOME.  Timed references will prune infrequently used
>>>>>>>>>>>> cache values, even though they're still strongly reachable.
>>>>>>>>>>>>
>>>>>>>>>>>> Unlike other cache libraries, custard apple only implements
>>>>>>>>>>>> reference capabilities, allowing the user to wrap this over
>> their
>>>>>>>>>>>> preferred collection, such as ConcurrentHashMap,
>>>>>>>>>>>> ConcurrentSkipListMap, NonBlockingHashMap etc, it reduces code
>>>>>>>>>>>> duplication and allows the user to benefit from new performance
>>>>>>>>>>>> improvements in popular collections.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Peter.
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/08/2015 8:57 PM, Patricia Shanahan wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Does it have a license that lets us do that?
>>>>>>>>>>>>> (If you are the writer, and copy it in yourself, it would be
>>>>>>>>>>>>> covered by your ICLA, just like any other code you contribute
>>>>>>>>>>>>> to Apache.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 8/12/2015 12:00 AM, Peter wrote: ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm a little busy right now to consider moving custard apple.
>>>>>>>>>>>>>> If you wan't you can always copy only the code in use into
>>>>>>>>>>>>>> org.apache.river.impl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>
>>