Failing tests with leftover tmp files from java minikdc

[Todd Lipcon <to...@cloudera.com>]

I just saw a release build precommit fail with the following:

*02:03:34* All tests passed, yet some left behind their test
output:*02:03:34* krb5kdc-1480470828522


http://104.196.14.100/job/kudu-gerrit/4926/BUILD_TYPE=RELEASE/consoleFull

It looks like this is left over from the MiniKDC in Java.

Are we being too strict about checking for leftover tmp stuff? Or is there
an easy fix to prevent this?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Failing tests with leftover tmp files from java minikdc

[Adar Dembo <ad...@cloudera.com>]

We've managed to make it this far (since commit 44fb5a9 in late 2014)
while being strict; I don't think we should relax it now.

After some digging, here's what's going on. TestMiniKuduCluster failed
once and was rerun. The second run passed, but the first failed in
such a way that it left the kdc's temp directory behind. From the
console output:

01:55:50 org.apache.kudu.client.TestMiniKuduCluster.testKerberos(org.apache.kudu.client.TestMiniKuduCluster)
01:55:50   Run 1: TestMiniKuduCluster.testKerberos:74 ? IO process
'kinit' failed: 1
01:55:50   Run 2: PASS

And the test output:

Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.137
sec <<< FAILURE! - in org.apache.kudu.client.TestMiniKuduCluster
testKerberos(org.apache.kudu.client.TestMiniKuduCluster)  Time
elapsed: 1.023 sec  <<< ERROR!
java.io.IOException: process 'kinit' failed: 1
        at org.apache.kudu.client.MiniKdc.checkReturnCode(MiniKdc.java:362)
        at org.apache.kudu.client.MiniKdc.kinit(MiniKdc.java:193)
        at org.apache.kudu.client.MiniKuduCluster.<init>(MiniKuduCluster.java:103)
        at org.apache.kudu.client.MiniKuduCluster.<init>(MiniKuduCluster.java:56)
        at org.apache.kudu.client.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:598)
        at org.apache.kudu.client.TestMiniKuduCluster.testKerberos(TestMiniKuduCluster.java:74)

Given that it was the constructor that threw an exception, I'm
guessing the try-with-resources in TestMiniKuduCluster doesn't bother
to close() the cluster? Dan, what do you think?

Separately, it's worth investigating why kinit failed. Maybe this was
the failure?

01:53:49.523 [DEBUG - Thread-8] (MiniKdc.java:353) kinit: Client
'testuser@KRBTEST.COM' not found in Kerberos database while getting
initial credentials

Perhaps kadmin.local finishes add_principal before it is actually
committed to its database?

On Wed, Nov 30, 2016 at 9:06 AM, Todd Lipcon <to...@cloudera.com> wrote:
> I just saw a release build precommit fail with the following:
>
> *02:03:34* All tests passed, yet some left behind their test
> output:*02:03:34* krb5kdc-1480470828522
>
>
> http://104.196.14.100/job/kudu-gerrit/4926/BUILD_TYPE=RELEASE/consoleFull
>
> It looks like this is left over from the MiniKDC in Java.
>
> Are we being too strict about checking for leftover tmp stuff? Or is there
> an easy fix to prevent this?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera