Re: Publishing the results of the TCKs

[Jim Driscoll <ji...@sun.com>]

Hi Sam -

I checked with the lawyers, and this was one of the things the is
specifically not allowed by the contract.

You may publish it on a password protected website, or email it to
either jcp@apache.org (all Apache members) or axis-tck@sml.apache.org
(who all have rights to view the TCK either from Apache or elsewhere),
but the sole thing you can say about TCK test status publicly is
"passed" or "failed".

Allow me to explain the reasoning here:  the thought is when you allow
people to publish full test results, you introduce the notion of "mostly
passing", i.e. "We pass 550 TCK tests out of 600!", when really, the
status is "Failed the TCK".

Yes, I know that that isn't Apache's intent in any way.  But the
contract's purpose is to guard against bad intent...

Hope that this isn't too noxious.

Jim

Sam Ruby wrote:
> 
> I was able to find [1], so I produced [2].  At the moment, it is
> password protected (user=axis, passwd=thanksJason).  What I want to know
> if *ANYBODY* has *ANY* concerns with publically publishing (i.e., on an
> apache.org site with no password) *ANY_PART_OF* the output the TCK when
> run against Axis?
> 
> Speak now, or forever hold your peace...
> 
> - Sam Ruby
> 
> [1] http://developer.java.sun.com/developer/technicalArticles/JCPtools2/
> [2] http://intertwingly.net/saajtck/report.html

Re: Publishing the results of the TCKs

[Sam Ruby <ru...@apache.org>]

Jim Driscoll wrote:
> 
> Unless, of course, Apache were willing to certify non-members as being
> covered by the license, which you aren't willing to do - so it's a
> problem created by both sides - Sun's desire for confidentiality, and
> Apache's reluctance to vouch for all her committers.

Vouch for what?  I can understand the argument that Apache needs to 
protect access to the TCK itself, but what is the argument for 
protectings committers from knowing when their changes break 
compatiblitity with the JSR?

Can't we find some middle ground here?

- Sam Ruby

Re: Publishing the results of the TCKs

[Jim Driscoll <ji...@sun.com>]

Sam Ruby wrote:
> 
> Jim Driscoll wrote:
>  > Hi Sam -
>  >
>  > I checked with the lawyers, and this was one of the things the is
>  > specifically not allowed by the contract.
> 
> This is very frustrating as you and I specifically talked about doing
> exactly this prior to the Apache/Sun agreement.
> 
> But somehow, this is not surprising, as it completes the cycle.  Apache
> sabre rattles.  There is much tension.  Agreement is reached just before
> the deadline.  Yet when it comes time for execution, the story is the
> same "my lawyer won't let me do it."

Actually, it's "your contract won't let you do it".  I'm sorry, but I
don't remember the conversation - I'm assuming it was before the
signing, where things went back on forth a bit.  But I talked with so
many people about this.  If I misled you in any way, or gave you
incorrect information, I'm truly sorry.  I do remember discussing this
at length with Jason, Chuck and a variety of lawyers.  In the end, the
confidentiality provisions were weakened, but not eliminated.  I
personally would have liked them eliminated, but there were good
arguments for allowing them to remain.

I assume you guys all discussed this at length internally as well, but I
was excluded from those discussions, so I couldn't really say.

>  > You may publish it on a password protected website, or email it to
>  > either jcp@apache.org (all Apache members) or axis-tck@sml.apache.org
>  > (who all have rights to view the TCK either from Apache or
>  > elsewhere), but the sole thing you can say about TCK test status
>  > publicly is "passed" or "failed".
> 
> Gee, thanks.

Figured you'd be thrilled :-(
 
>  > Allow me to explain the reasoning here:  the thought is when you
>  > allow people to publish full test results, you introduce the notion
>  > of "mostly passing", i.e. "We pass 550 TCK tests out of 600!", when
>  > really, the status is "Failed the TCK".
>  >
>  > Yes, I know that that isn't Apache's intent in any way.  But the
>  > contract's purpose is to guard against bad intent...
>  >
>  > Hope that this isn't too noxious.
> 
> First thing I would like to clearly establish is that this is a
> requirement of Sun and/or the leads for these specifications, and not of
> the JCP itself, right?  After all, IIRC the business terms for JSR 110
> are as follows:

<snip>

Yep, it's a requirement of the contract signed between Sun and Apache. 
AFAIK, it's not a JCP requirement.  However, I'm not the JCP expert. 
Jason / Chuck are more likely to have the definitive answer there, or I
can check internally if they're unavailable.  I'd have to refer to the
JSPA verbiage, and that's still a work in progress anyway.

> Now let me explain what I was trying to accomplish.  At the present
> time, we pass 100% of the SAA JTCK, and are within two tests of passing
> the JAXRPC TCK.  What I was hoping to do was to set things up so that
> the tests ran automatically and generally successfully.  But at the
> point where somebody - any contributor, really - broke compatibility in
> a nightly build, they would be notified of that fact, generally within
> 24 hours.
> 
> Instead, what is being proposed is that unless they are one of the
> select few, not only can't they run these tests themselves, they can't
> even directly view the results of their efforts.
> 
> So, yes, I view that as noxious.

Unless, of course, Apache were willing to certify non-members as being
covered by the license, which you aren't willing to do - so it's a
problem created by both sides - Sun's desire for confidentiality, and
Apache's reluctance to vouch for all her committers.

One solution that I can think of is to have the people who are active
committers apply to the board for license permission.  Also noxious, but
it does what you want, eventually.

BTW - congratulations on the team's progress!  Almost there.

Jim

Re: Publishing the results of the TCKs

[Sam Ruby <ru...@apache.org>]

Jim Driscoll wrote:
 > Hi Sam -
 >
 > I checked with the lawyers, and this was one of the things the is
 > specifically not allowed by the contract.

This is very frustrating as you and I specifically talked about doing
exactly this prior to the Apache/Sun agreement.

But somehow, this is not surprising, as it completes the cycle.  Apache
sabre rattles.  There is much tension.  Agreement is reached just before
the deadline.  Yet when it comes time for execution, the story is the
same "my lawyer won't let me do it."

 > You may publish it on a password protected website, or email it to
 > either jcp@apache.org (all Apache members) or axis-tck@sml.apache.org
 > (who all have rights to view the TCK either from Apache or
 > elsewhere), but the sole thing you can say about TCK test status
 > publicly is "passed" or "failed".

Gee, thanks.

 > Allow me to explain the reasoning here:  the thought is when you
 > allow people to publish full test results, you introduce the notion
 > of "mostly passing", i.e. "We pass 550 TCK tests out of 600!", when
 > really, the status is "Failed the TCK".
 >
 > Yes, I know that that isn't Apache's intent in any way.  But the
 > contract's purpose is to guard against bad intent...
 >
 > Hope that this isn't too noxious.

First thing I would like to clearly establish is that this is a
requirement of Sun and/or the leads for these specifications, and not of
the JCP itself, right?  After all, IIRC the business terms for JSR 110
are as follows:

 > The TCK will be open source and be released with the Common Public
 > License (CPL). Making it open increases the community understanding
 > of the API since the source provides more visibility to the test
 > coverage. Availability of the source will help reference implementers
 > understand the test requirements better. Over time this will lead to
 > a better implementation. Even though the source is open, there is a
 > maintainer of the official TCK, the JSR Maintenance Spec Lead. Open
 > Source allows others to 'submit' changes to the TCK, but doesn't
 > guarantee that they are approved. Compliance will only be by passing
 > the TCK approved by the committee appropriate to the approved API.
 > Compliance will be claimed via self-certification. In a true open
 > source environment, customers, analysts, consultants, governmental
 > agencies, etc. will all have access to the TCK to see how it works
 > and run implementations through it. Violators of the
 > self-certification and logo use rules will be quickly exposed. This
 > "Market Driven" compliance is far more powerful than trying to
 > maintain control through onerous legal conditions and exceptions.
 > There are no separate business issues involved in providing JSR110
 > certification. Self-certification is the rule. Enforcement is simple.
 >  If after JSR110 is approved, someone claims compliance and any user
 > finds otherwise, the TCK provides a relatively objective way of
 > proving noncompliance."

Now let me explain what I was trying to accomplish.  At the present
time, we pass 100% of the SAA JTCK, and are within two tests of passing
the JAXRPC TCK.  What I was hoping to do was to set things up so that
the tests ran automatically and generally successfully.  But at the
point where somebody - any contributor, really - broke compatibility in 
a nightly build, they would be notified of that fact, generally within 
24 hours.

Instead, what is being proposed is that unless they are one of the
select few, not only can't they run these tests themselves, they can't
even directly view the results of their efforts.

So, yes, I view that as noxious.

 > Jim

- Sam Ruby