JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Randall Hauch <rh...@gmail.com>]

First of all, I'd like to thank the whole Jackrabbit community for maintaining and adding to the "jackrabbit-jcr-tests" suite of unit tests. IIUC, these tests started out as the suite that is used by the JSR-283 TCK, but which now include quite a few fixes and enhancements. Having these tests is a huge plus for Jackrabbit but also for other JCR implementations (including ModeShape [1][2] and Jackrabbit's own Oak project [3]), since this large suite of independent tests helps any implementation to provide behavior consistent with the specification.

The official JSR-283 TCK tests [4] was released in August 2009, and IIUC was based on one of the "jackrabbit-jcr-tests" module in one of the 2.0.x releases of Jackrabbit. Since then, the official TCK hasn't been updated or corrected, while the "jackrabbit-jcr-tests" have undergone quite a few corrections and changes. (I recently learned of this subtle distinction.)

Specifically, since the 2.0 release of Jackrabbit, all of the following Jackrabbit issues changed some part of the unit tests in "jackrabbit-jcr-tests" [5]:

JCR-3316: fix namespace name to be an absolute URI
JCR-3313 Changed a ColumnTest test to only check for required columns
JCR-3307: check supported option OPTION_ACTIVITIES_SUPPORTED
JCR-3301: fix typos in test method names and associated config files
JCR-3302: add missing refresh() call on test session
JCR-3300: make RSessionAccessControlTest extend the proper base class so that the repo option check happens
JCR-3207: add TCK test for Info map of NODE_MOVED event on node reordering
JCR-3205 - Missing support for lock timeout and ownerHint in jcr-server
JCR-3196: reduce deprecation warning noise in TCK testsReduce warning noise
JCR-3195: fix assumptions about null-ness of lock token
JCR-3177: Remove jdk 1.4 restriction for jcr-tests
JCR-3051: AbstractLockTest.testLockExpiration fails intermittently
JCR-2903: Session.importXml should close the input stream (as to JSR 283/JCR 2.0)
JCR-2756: Shareable node test failures
JCR-2719: Incorrect outer join TCK tests
JCR-2693: Logging per test case
JCR-2665: JCR Test for Adding Node Type Tests That Abstract Nodes Can Be Added as Children, contrary to JCR 2.0 specification
JCR-2663: JCR unit tests use invalid queries
JCR-2662: check for repository support of journaled observation, otherwise skip test
JCR-2648: PropertyImpl.getNode() and NamePropertyTest use different exception than documented in the JCR API JavaDoc
JCR-2536: spi2davex: InvalidItemStateException not properly extracted from ambiguous response error 
JCR-2565: spi2dav: Overwrite header T specified for MOVE and COPY causes
JCR-2515: ISO8601 uses default DecimalFormat constructor using locale specific digits
JCR-2490: jackrabbit wrongly think nodetype is changed on nodetype re-registration
JCR-2104: JSR 283 Full Versioning - checkpoint must be aware of current activity for the checkout part
JCR-515: Enhance test data

IIUC most/all of them are still unfixed in the official TCK, and this can cause quite a bit of grief for other implementations. Plain and simple, the "jackrabbit-jcr-tests" module contains tests that are more accurately verifying the expectations of the JSR-283 specification than the official TCK. And this is why ModeShape adopted has for years used the "jackrabbit-jcr-tests" in its builds, but in doing so we've run into some challenges.

Since the "jackrabbit-jcr-tests" module is part of the Jackrabbit codebase, it is released along with the rest of Jackrabbit. Of course, this makes perfect sense for the Jackrabbit community. But considering that the "jackrabbit-jcr-tests" are used by at least two other OSS projects as essentially a "more correct and maintained" proxy for the TCK tests, this inclusion in the Jackrabbit releases means the TCK tests are not as frequently released as we'd hope.

Is there any interest in separating the "jackrabbit-jcr-tests"? I'm not sure if that means a completely separate (sub)project, or whether there are better alternatives (e.g., a simple project on GitHub?). Separating the tests would have a number of benefits:
The "TCK unit tests" could be released on their own schedule, with version numbers that are semantically related to the changes made within that release
The JIRA issues would better reflect the work that's been done (e.g., release notes), the number of changes, and the outstanding issues (see below)
Emphasis can be placed on encouraging reuse and participation by other projects, and ensuring that the tests are as correct as possible
This can possibly form a basis for either updating the official TCK (if that's even possible) or providing a form that excludes/disables the incorrect tests and includes fixed tests (per the TCK appeals process), making the process much easier for those submitting results (with appeals) as well as those reviewing/approving results.
Perhaps allow for more tests to be added, though this may be difficult if these unit tests do form the basis for an updated official TCK. (ModeShape has a number of tests that we haven't yet submitted, since we thought until recently that the official TCK was periodically updated with the latest "jackrabbit-jcr-tests".)  

As mentioned, there are currently 10 others issues that are still open, in progress, or have a patch available [6]:


JCR-3321 Strange XPath query in OrderByMultiTypeTest.testMultipleOrder
JCR-3302 Remove incorrect assumptions with respect to visibility of changes done by other sessions
JCR-3301 Remove incorrect test assumptions
JCR-3196 reduce deprecation warning noise in TCK tests
JCR-3096 AbstractObservationTest does not cover JCR 2.0 functionality
JCR-2756 Shareable node test failures
JCR-2661 Two JCR unit tests expect delete events for nodes under deleted node, contrary to JCR 2.0 specification
JCR-2348 Exclude problematic node types from tests
JCR-3322 add TCK coverage of isNodeType(expandedName)
JCR-2666 JCR TCK Test for Restoring Version Tests That Versionable Child Is also Restored, contrary to JCR 2.0 specification


(JSR-1991 is also still open and in the report, but has to do more with the OSGi bundling of the JAR.)

I have no doubt that at a minimum the Jackrabbit, ModeShape, and Oak communities would very much like to get these resolved and would continue to work together to build upon, improve and maintain the JCR unit tests. But I believe that this will be even better if the JCR unit tests can be separated.

Thanks for considering, and I look forward to discussing the possibilities.

Best regards,

Randall Hauch
Project lead, ModeShape
http://modeshape.org


[1] http://modeshape.org
[2] https://docs.jboss.org/author/display/MODE/Home
[3] http://wiki.apache.org/jackrabbit/Jackrabbit%203
[4] http://www.day.com/day/en/products/jcr/jsr-283.html
[5] https://github.com/apache/jackrabbit/commits/trunk/jackrabbit-jcr-tests/src
[6] https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+JCR+AND+component+%3D+jackrabbit-jcr-tests+AND+status+in+%28Open%2C+%22In+Progress%22%2C+%22Patch+Available%22%29+ORDER+BY+status+ASC%2C+key+DESC

Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Randall Hauch <rh...@gmail.com>]

Thanks, Jukka. See below for a few responses/questions. As always, thanks again!


On Friday, August 3, 2012 at 11:51 AM, Jukka Zitting wrote:

> Hi,
> 
> On Fri, Aug 3, 2012 at 5:21 PM, Randall Hauch <rhauch@gmail.com (mailto:rhauch@gmail.com)> wrote:
> > The Jackrabbit TCK page [2] states that Jackrabbit's JCR Tests module is
> > used within the official JSR-283 TCK [3] but is not the official TCK test.
> > Obviously the former have been maintained and continue to improve, but the
> > official JSR-283 TCK tests appear to have been released last on August 19,
> > 2009. Is it possible that the official JSR-283 TCK tests can be updated, and
> > would having a separate release schedule help with this in any way?
> > 
> 
> 
> The only way to update the official JSR-283 TCK is through the JCP
> maintenance process [1].
> 
> > (If the official TCK tests cannot be updated, then I presume any project wanting
> > to claim JSR-283 compliance would have to run the official TCKs and appeal each
> > of the older incorrect tests by stating the issue and perhaps using the
> > updated/corrected tests Jackrabbit JCR Tests module as "corrections".
> > 
> 
> 
> Yes, the appeal process is documented in [2] and relies on an exclude
> list of buggy tests maintained by the spec lead. In practice, if you
> have pointers to relevant jackrabbit-jcr-tests bug reports and can
> demonstrate that your implementation passes the test after that issue
> is fixed, the spec lead will be happy to put the test case on the
> exclude list for you.
> 
> 

What is the preferred way to demonstrate that our implementation passes all of the JCR Tests? We obviously run them in our builds, but we could also set up a separate Maven project to do that. I presume there's no way to easily run the official TCK tests but forcibly use the latest unofficial JCR Tests. Is there a preferred point of contact, other than jcr@day.com ?
 
> 
> > Is there anything that makes this easier with a non-updated TCK?
> 
> 
> It's a more light-weight process than doing a maintenance release through JCP.
> 
> > How does Jackrabbit show spec and TCK compliance?
> 
> We rely on the appeals process and exclude list as described above. In
> practice that means that we're good as long as Jackrabbit passes the
> latest version of the tests in jackrabbit-jcr-tests.
> 
> > The official TCK and Jackrabbit's JCR tests unsurprisingly do not completely
> > cover the specification, and doing so would require a significant amount of
> > effort. However, there may be opportunity over time for projects to propose
> > adding new tests.
> > 
> 
> 
> Agreed. Especially with multiple active JCR implementations I think
> there's a shared incentive to maintain compatibility beyond that of
> what the official TCK tests. There's still plenty of place for healthy
> competition on performance, scalability, maintainability and various
> other features and "-ilities" that fall outside the scope of JCR.
> 
> Thinking further, a standalone test codebase would also be a great
> place to cooperate on things like performance benchmarks or other
> tests that go beyond the scope of the JCR spec but that would still
> add value to implementors of content repositories.
> 
> 

It would be great to collaborate on this. We also have the start of a performance test framework (as does Jackrabbit) that requires nothing but Maven and a JDK; the set of tests is minimal.
> 
> > If the JCR Tests' release cycle were independent from the Jackrabbit's
> > release cycle, then the process of adding new tests and releasing updated
> > JCR tests might be easier. On the other hand, verification that the tests are
> > valid may become a bit harder.
> > 
> 
> 
> Not necessarily. If the tests were maintained outside the main
> Jackrabbit trunk, it would be easier set up a CI build that runs all
> updates to the TCK codebase against the official JCR RI and latest
> stable versions of Jackrabbit, ModeShape and other JCR implementations
> (with excludes for tests that are known issues for each particular
> implementation). That should make it much easier than now to catch
> problems where the TCK codebase makes assumptions based on just a
> single implementation.
> 
> 

Setting up CI to automatically test against the latest stable releases is a great idea.
 
> 
> > For example, the ModeShape project has around 70 additional tests [4] that
> > we'd be willing to donate. Most of these are around verifying administration
> > privileges (e.g., registering namespaces, node types, etc., especially for
> > anonymous users), versioning, and locking.
> > 
> 
> 
> That would be awesome!
> 
> We also have some extra generic JCR tests, written for the jcr2spi
> component, that could and ideally should be used as a part of the TCK.
> 
We have a lot of query-related tests, too. But those are more basic JUnit tests that don't extend the AbstractJcrTest base class. It'd take a bit more work, but we could probably easily convert them to something compatible with JCR Tests.
 
> 
> > Additionally, the Oak effort is already running the JCR tests and has
> > recently found/fixed issues, too. Can those participating in Oak offer how
> > much they might expect to simply reuse vs. add new tests?
> > 
> 
> 
> 

> For now our goal is just to pass the already existing TCK tests, but
> I'm sure that over time we'll encounter cases where existing clients
> that assume only standard JCR functionality fail on Oak because of
> some problems in our code. Capturing such cases as TCK tests would be
> quite useful.
> 
> [1] http://jcp.org/en/procedures/jcp2#5
> [2] http://www.day.com/content/day/en/products/jcr/jsr-283/_jcr_content/par/download_1/file.res/jsr-283-tck-appeal.pdf
> 
> BR,
> 
> Jukka Zitting 

Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Aug 3, 2012 at 5:21 PM, Randall Hauch <rh...@gmail.com> wrote:
> The Jackrabbit TCK page [2] states that Jackrabbit's JCR Tests module is
> used within the official JSR-283 TCK [3] but is not the official TCK test.
> Obviously the former have been maintained and continue to improve, but the
> official JSR-283 TCK tests appear to have been released last on August 19,
> 2009. Is it possible that the official JSR-283 TCK tests can be updated, and
> would having a separate release schedule help with this in any way?

The only way to update the official JSR-283 TCK is through the JCP
maintenance process [1].

> (If the official TCK tests cannot be updated, then I presume any project wanting
> to claim JSR-283 compliance would have to run the official TCKs and appeal each
> of the older incorrect tests by stating the issue and perhaps using the
> updated/corrected tests Jackrabbit JCR Tests module as "corrections".

Yes, the appeal process is documented in [2] and relies on an exclude
list of buggy tests maintained by the spec lead. In practice, if you
have pointers to relevant jackrabbit-jcr-tests bug reports and can
demonstrate that your implementation passes the test after that issue
is fixed, the spec lead will be happy to put the test case on the
exclude list for you.

> Is there anything that makes this easier with a non-updated TCK?

It's a more light-weight process than doing a maintenance release through JCP.

> How does Jackrabbit show spec and TCK compliance?

We rely on the appeals process and exclude list as described above. In
practice that means that we're good as long as Jackrabbit passes the
latest version of the tests in jackrabbit-jcr-tests.

> The official TCK and Jackrabbit's JCR tests unsurprisingly do not completely
> cover the specification, and doing so would require a significant amount of
> effort. However, there may be opportunity over time for projects to propose
> adding new tests.

Agreed. Especially with multiple active JCR implementations I think
there's a shared incentive to maintain compatibility beyond that of
what the official TCK tests. There's still plenty of place for healthy
competition on performance, scalability, maintainability and various
other features and "-ilities" that fall outside the scope of JCR.

Thinking further, a standalone test codebase would also be a great
place to cooperate on things like performance benchmarks or other
tests that go beyond the scope of the JCR spec but that would still
add value to implementors of content repositories.

> If the JCR Tests' release cycle were independent from the Jackrabbit's
> release cycle, then the process of adding new tests and releasing updated
> JCR tests might be easier. On the other hand, verification that the tests are
> valid may become a bit harder.

Not necessarily. If the tests were maintained outside the main
Jackrabbit trunk, it would be easier set up a CI build that runs all
updates to the TCK codebase against the official JCR RI and latest
stable versions of Jackrabbit, ModeShape and other JCR implementations
(with excludes for tests that are known issues for each particular
implementation). That should make it much easier than now to catch
problems where the TCK codebase makes assumptions based on just a
single implementation.

> For example, the ModeShape project has around 70 additional tests [4] that
> we'd be willing to donate. Most of these are around verifying administration
> privileges (e.g., registering namespaces, node types, etc., especially for
> anonymous users), versioning, and locking.

That would be awesome!

We also have some extra generic JCR tests, written for the jcr2spi
component, that could and ideally should be used as a part of the TCK.

> Additionally, the Oak effort is already running the JCR tests and has
> recently found/fixed issues, too. Can those participating in Oak offer how
> much they might expect to simply reuse vs. add new tests?

For now our goal is just to pass the already existing TCK tests, but
I'm sure that over time we'll encounter cases where existing clients
that assume only standard JCR functionality fail on Oak because of
some problems in our code. Capturing such cases as TCK tests would be
quite useful.

[1] http://jcp.org/en/procedures/jcp2#5
[2] http://www.day.com/content/day/en/products/jcr/jsr-283/_jcr_content/par/download_1/file.res/jsr-283-tck-appeal.pdf

BR,

Jukka Zitting

Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Randall Hauch <rh...@gmail.com>]

Hi, Julian:

I agree that the TCK unit tests are in really good shape, especially given that all of our known issues with them will be addressed in the 2.5.1 release (see [1]). Separating the TCK unit tests with its own independent release cycle will definitely require some up front work, and whether this work will be worth it may depend on several things:

* Relationship to the official TCK

The Jackrabbit TCK page [2] states that Jackrabbit's JCR Tests module is used within the official JSR-283 TCK [3] but is not the official TCK test. Obviously the former have been maintained and continue to improve, but the official JSR-283 TCK tests appear to have been released last on August 19, 2009. Is it possible that the official JSR-283 TCK tests can be updated, and would having a separate release schedule help with this in any way? (If the official TCK tests cannot be updated, then I presume any project wanting to claim JSR-283 compliance would have to run the official TCKs and appeal each of the older incorrect tests by stating the issue and perhaps using the updated/corrected tests Jackrabbit JCR Tests module as "corrections". Is there anything that makes this easier with a non-updated TCK? How does Jackrabbit show spec and TCK compliance?)

* New tests

The official TCK and Jackrabbit's JCR tests unsurprisingly do not completely cover the specification, and doing so would require a significant amount of effort. However, there may be opportunity over time for projects to propose adding new tests. If the JCR Tests' release cycle were independent from the Jackrabbit's release cycle, then the process of adding new tests and releasing updated JCR tests might be easier. On the other hand, verification that the tests are valid may become a bit harder. (We'd probably want to define a process to govern such additions and whether they are valid. At the moment, the )

For example, the ModeShape project has around 70 additional tests [4] that we'd be willing to donate. Most of these are around verifying administration privileges (e.g., registering namespaces, node types, etc., especially for anonymous users), versioning, and locking. Obviously we'd do this by submitting a proposed patch for the appropriate JCR unit test cases, and we'd want others to verify the expectations are indeed mandated in the specification. (Any Jackrabbit failures of the proposed tests might be a signal the tests.

Additionally, the Oak effort is already running the JCR tests and has recently found/fixed issues, too. Can those participating in Oak offer how much they might expect to simply reuse vs. add new tests?


Best regards,

Randall Hauch
Project lead, ModeShape

[1] http://markmail.org/thread/isihzvirvrc4idpp
[2] http://jackrabbit.apache.org/jackrabbit-jcr-tests.html
[3] http://www.day.com/day/en/products/jcr/jsr-283.html
[4] https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/test/java/org/modeshape/jcr/ModeShapeTckTest.java


On Thursday, June 7, 2012 at 11:58 AM, Julian Reschke wrote:

> On 2012-06-07 18:19, Michael Dürig wrote:
> > Hi Randall,
> >  
> > I think forking of the jcr-test module into its own release cycle
> > definitely makes sense. The easiest thing would be to just make it a
> > sub-project of Jackrabbit and move it from trunk/jackrabbit-jcr-tests to
> > jcr-tests/trunk. In addition we'd probably also need a separate issued
> > tracker and migrate existing issues.
> >  
> > However, this comes with some additional effort and complexity. Apart
> > from the initial move and fixing of existing dependencies we'd need a
> > release manager who takes care of the new sub-project. Not sure whether
> > Alex has enough cylces left to take this over.
> >  
> > Let's see what others think of this and then open an issue from the
> > conclusion.
> >  
> > Michael
> > ...
> >  
>  
>  
> I agree that decoupling is the right thing to do. On the other hand,  
> decoupling also requires (a) some initial work, and (b) ongoing work to  
> actually cut the separate releases.
>  
> It seems that the original problem that triggered this discussion was  
> the amount of changes in trunk that were not present in a non-SNAPSHOT  
> release. In the meantime, 2.5.0 *has* been released, and looking at the  
> Jackrabbit release history, we've made releases every few months.
>  
> So is the trouble really worth it? Who volunteers to take case of this?
>  
> Best regards, Julian  

Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Julian Reschke <ju...@gmx.de>]

On 2012-06-07 18:19, Michael Dürig wrote:
> Hi Randall,
>
> I think forking of the jcr-test module into its own release cycle
> definitely makes sense. The easiest thing would be to just make it a
> sub-project of Jackrabbit and move it from trunk/jackrabbit-jcr-tests to
> jcr-tests/trunk. In addition we'd probably also need a separate issued
> tracker and migrate existing issues.
>
> However, this comes with some additional effort and complexity. Apart
> from the initial move and fixing of existing dependencies we'd need a
> release manager who takes care of the new sub-project. Not sure whether
> Alex has enough cylces left to take this over.
>
> Let's see what others think of this and then open an issue from the
> conclusion.
>
> Michael
> ...

I agree that decoupling is the right thing to do. On the other hand, 
decoupling also requires (a) some initial work, and (b) ongoing work to 
actually cut the separate releases.

It seems that the original problem that triggered this discussion was 
the amount of changes in trunk that were not present in a non-SNAPSHOT 
release. In the meantime, 2.5.0 *has* been released, and looking at the 
Jackrabbit release history, we've made releases every few months.

So is the trouble really worth it? Who volunteers to take case of this?

Best regards, Julian

Re: JSR-283 official TCK and the 'jackrabbit-jcr-tests'

[Michael Dürig <md...@apache.org>]

Hi Randall,

I think forking of the jcr-test module into its own release cycle 
definitely makes sense. The easiest thing would be to just make it a 
sub-project of Jackrabbit and move it from trunk/jackrabbit-jcr-tests to 
jcr-tests/trunk. In addition we'd probably also need a separate issued 
tracker and migrate existing issues.

However, this comes with some additional effort and complexity. Apart 
from the initial move and fixing of existing dependencies we'd need a 
release manager who takes care of the new sub-project. Not sure whether 
Alex has enough cylces left to take this over.

Let's see what others think of this and then open an issue from the 
conclusion.

Michael


On 7.6.12 15:00, Randall Hauch wrote:
> First of all, I'd like to thank the whole Jackrabbit community for
> maintaining and adding to the "jackrabbit-jcr-tests" suite of unit
> tests. IIUC, these tests started out as the suite that is used by the
> JSR-283 TCK, but which now include quite a few fixes and enhancements.
> Having these tests is a huge plus for Jackrabbit but also for other JCR
> implementations (including ModeShape [1][2] and Jackrabbit's own Oak
> project [3]), since this large suite of independent tests helps any
> implementation to provide behavior consistent with the specification.
>
> The official JSR-283 TCK tests [4] was released in August 2009, and IIUC
> was based on one of the "jackrabbit-jcr-tests" module in one of the
> 2.0.x releases of Jackrabbit. Since then, the official TCK hasn't been
> updated or corrected, while the "jackrabbit-jcr-tests" have undergone
> quite a few corrections and changes. (I recently learned of this subtle
> distinction.)
>
> Specifically, since the 2.0 release of Jackrabbit, all of the following
> Jackrabbit issues changed some part of the unit tests in
> "jackrabbit-jcr-tests" [5]:
>
> JCR-3316: fix namespace name to be an absolute URI
> JCR-3313 Changed a ColumnTest test to only check for required columns
> JCR-3307: check supported option OPTION_ACTIVITIES_SUPPORTED
> JCR-3301: fix typos in test method names and associated config files
> JCR-3302: add missing refresh() call on test session
> JCR-3300: make RSessionAccessControlTest extend the proper base class so
> that the repo option check happens
> JCR-3207: add TCK test for Info map of NODE_MOVED event on node reordering
> JCR-3205 - Missing support for lock timeout and ownerHint in jcr-server
> JCR-3196: reduce deprecation warning noise in TCK testsReduce warning noise
> JCR-3195: fix assumptions about null-ness of lock token
> JCR-3177: Remove jdk 1.4 restriction for jcr-tests
> JCR-3051: AbstractLockTest.testLockExpiration fails intermittently
> JCR-2903: Session.importXml should close the input stream (as to JSR
> 283/JCR 2.0)
> JCR-2756: Shareable node test failures
> JCR-2719: Incorrect outer join TCK tests
> JCR-2693: Logging per test case
> JCR-2665: JCR Test for Adding Node Type Tests That Abstract Nodes Can Be
> Added as Children, contrary to JCR 2.0 specification
> JCR-2663: JCR unit tests use invalid queries
> JCR-2662: check for repository support of journaled observation,
> otherwise skip test
> JCR-2648: PropertyImpl.getNode() and NamePropertyTest use different
> exception than documented in the JCR API JavaDoc
> JCR-2536: spi2davex: InvalidItemStateException not properly extracted
> from ambiguous response error
> JCR-2565: spi2dav: Overwrite header T specified for MOVE and COPY causes
> JCR-2515: ISO8601 uses default DecimalFormat constructor using locale
> specific digits
> JCR-2490: jackrabbit wrongly think nodetype is changed on nodetype
> re-registration
> JCR-2104: JSR 283 Full Versioning - checkpoint must be aware of current
> activity for the checkout part
> JCR-515: Enhance test data
>
> IIUC most/all of them are still unfixed in the official TCK, and this
> can cause quite a bit of grief for other implementations. Plain and
> simple, the "jackrabbit-jcr-tests" module contains tests that are more
> accurately verifying the expectations of the JSR-283 specification than
> the official TCK. And this is why ModeShape adopted has for years used
> the "jackrabbit-jcr-tests" in its builds, but in doing so we've run into
> some challenges.
>
> Since the "jackrabbit-jcr-tests" module is part of the Jackrabbit
> codebase, it is released along with the rest of Jackrabbit. Of course,
> this makes perfect sense for the Jackrabbit community. But considering
> that the "jackrabbit-jcr-tests" are used by at least two other OSS
> projects as essentially a "more correct and maintained" proxy for the
> TCK tests, this inclusion in the Jackrabbit releases means the TCK tests
> are not as frequently released as we'd hope.
>
> Is there any interest in separating the "jackrabbit-jcr-tests"? I'm not
> sure if that means a completely separate (sub)project, or whether there
> are better alternatives (e.g., a simple project on GitHub?). Separating
> the tests would have a number of benefits:
>
>  1. The "TCK unit tests" could be released on their own schedule, with
>     version numbers that are semantically related to the changes made
>     within that release
>  2. The JIRA issues would better reflect the work that's been done
>     (e.g., release notes), the number of changes, and the outstanding
>     issues (see below)
>  3. Emphasis can be placed on encouraging reuse and participation by
>     other projects, and ensuring that the tests are as correct as possible
>  4. This can possibly form a basis for either updating the official TCK
>     (if that's even possible) or providing a form that excludes/disables
>     the incorrect tests and includes fixed tests (per the TCK appeals
>     process), making the process much easier for those submitting
>     results (with appeals) as well as those reviewing/approving results.
>  5. Perhaps allow for more tests to be added, though this may be
>     difficult if these unit tests do form the basis for an updated
>     official TCK. (ModeShape has a number of tests that we haven't yet
>     submitted, since we thought until recently that the official TCK was
>     periodically updated with the latest "jackrabbit-jcr-tests".)
>
> As mentioned, there are currently 10 others issues that are still open,
> in progress, or have a patch available [6]:
>
> JCR-3321Strange XPath query in OrderByMultiTypeTest.testMultipleOrder
> JCR-3302Remove incorrect assumptions with respect to visibility of
> changes done by other sessions
> JCR-3301Remove incorrect test assumptions
> JCR-3196reduce deprecation warning noise in TCK tests
> JCR-3096AbstractObservationTest does not cover JCR 2.0 functionality
> JCR-2756Shareable node test failures
> JCR-2661Two JCR unit tests expect delete events for nodes under deleted
> node, contrary to JCR 2.0 specification
> JCR-2348Exclude problematic node types from tests
> JCR-3322add TCK coverage of isNodeType(expandedName)
> JCR-2666JCR TCK Test for Restoring Version Tests That Versionable Child
> Is also Restored, contrary to JCR 2.0 specification
> (JSR-1991 is also still open and in the report, but has to do more with
> the OSGi bundling of the JAR.)
>
> I have no doubt that at a minimum the Jackrabbit, ModeShape, and Oak
> communities would very much like to get these resolved and would
> continue to work together to build upon, improve and maintain the JCR
> unit tests. But I believe that this will be even better if the JCR unit
> tests can be separated.
>
> Thanks for considering, and I look forward to discussing the possibilities.
>
> Best regards,
>
> Randall Hauch
> Project lead, ModeShape
> http://modeshape.org
>
>
> [1] http://modeshape.org
> [2] https://docs.jboss.org/author/display/MODE/Home
> [3] http://wiki.apache.org/jackrabbit/Jackrabbit%203
> [4] http://www.day.com/day/en/products/jcr/jsr-283.html
> [5]
> https://github.com/apache/jackrabbit/commits/trunk/jackrabbit-jcr-tests/src
> [6]
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+JCR+AND+component+%3D+jackrabbit-jcr-tests+AND+status+in+%28Open%2C+%22In+Progress%22%2C+%22Patch+Available%22%29+ORDER+BY+status+ASC%2C+key+DESC
> <https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+JCR+AND+component+%3D+jackrabbit-jcr-tests+AND+status+in+%28Open%2C+%22In+Progress%22%2C+%22Patch+Available%22%29+ORDER+BY+status+ASC%2C+key+DESC>
>