You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org> on 2025/09/16 18:46:45 UTC
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23433
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Deflake testRandomBackupAndRestore
The backup/restore tests were intermittently failing. This wasn’t
obvious at first because dist-test archives contained empty
_ARCHIVE_TOO_BIG_-* files.
Root cause was a Base64 dependency mismatch:
java.lang.NoClassDefFoundError: org/apache/commons/net/util/Base64
Switching from org.apache.commons.net.util.Base64 to java.util.Base64
resolves the failure. The issue is reproduced by the new test
testBinaryColumnBackupReproducesBase64Issue.
Running the repro test without the fix results in this stacktrace:
13:45:05.130 [Test worker] ERROR org.apache.kudu.test.junit.RetryRule -- org.apache.kudu.backup.TestKuduBackup.testBinaryColumnBackupReproducesBase64Issue: failed attempt 1
java.lang.NoClassDefFoundError: org/apache/commons/net/util/Base64
at org.apache.kudu.backup.TableMetadata$.valueToString(TableMetadata.scala:351)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getTableMetadata$1(TableMetadata.scala:78)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getTableMetadata(TableMetadata.scala:61)
at org.apache.kudu.backup.KuduBackup$.doBackup(KuduBackup.scala:102)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$6(KuduBackup.scala:146)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$5(KuduBackup.scala:146)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:66)
at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1064)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1061)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:968)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:963)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.net.util.Base64
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 56 common frames omitted
During investigation I found another bug with BINARY range-partition
boundaries causing:
java.lang.ClassCastException: class java.nio.HeapByteBuffer cannot be
cast to class [B
This is fixed in the same valueToString switch; covered by
testBinaryPartitionBoundariesBase64Issue.
Running the repro test without the fix results in this stacktrace:
13:39:47.053 [ForkJoinPool-2-worker-1] ERROR org.apache.kudu.backup.KuduBackup$ -- Failed to back up table binary-partition-base64-issue
java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B
at org.apache.kudu.backup.TableMetadata$.valueToString(TableMetadata.scala:351)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getBoundValues$2(TableMetadata.scala:225)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getBoundValues(TableMetadata.scala:219)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getRangePartitionMetadata$2(TableMetadata.scala:196)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getRangePartitionMetadata(TableMetadata.scala:195)
at org.apache.kudu.backup.TableMetadata$.getPartitionSchemaMetadata(TableMetadata.scala:125)
at org.apache.kudu.backup.TableMetadata$.getTableMetadata(TableMetadata.scala:106)
at org.apache.kudu.backup.KuduBackup$.doBackup(KuduBackup.scala:102)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$6(KuduBackup.scala:146)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$5(KuduBackup.scala:146)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:66)
at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1064)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1061)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:968)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:963)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Before these fixes, running org.apache.kudu.backup.TestKuduBackup 100
times yielded 43 failures / 57 passes. After the fixes: 100/100 passes.
The random nature of test failures is because the problematic test:
testRandomBackupAndRestore uses randomized schema and data.
This is not a full RCA I wanted to deflake with reasonable solution;
I’ve opened KUDU-3694 to track root-cause follow-ups and additional
action items.
Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Reviewed-on: http://gerrit.cloudera.org:8080/23358
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Reviewed-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Gabriella Lotz <lo...@gmail.com>
Tested-by: Marton Greber <gr...@gmail.com>
(cherry picked from commit 724911497f7b2b2d6bc53f1c510ac0b82889bbb3)
---
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
2 files changed, 153 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/33/23433/1
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23433 )
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/1809/
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Sep 2025 18:47:02 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23433 )
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/1809/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Sep 2025 20:16:13 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23433 )
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Sep 2025 23:39:51 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23433 )
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Deflake testRandomBackupAndRestore
The backup/restore tests were intermittently failing. This wasn’t
obvious at first because dist-test archives contained empty
_ARCHIVE_TOO_BIG_-* files.
Root cause was a Base64 dependency mismatch:
java.lang.NoClassDefFoundError: org/apache/commons/net/util/Base64
Switching from org.apache.commons.net.util.Base64 to java.util.Base64
resolves the failure. The issue is reproduced by the new test
testBinaryColumnBackupReproducesBase64Issue.
Running the repro test without the fix results in this stacktrace:
13:45:05.130 [Test worker] ERROR org.apache.kudu.test.junit.RetryRule -- org.apache.kudu.backup.TestKuduBackup.testBinaryColumnBackupReproducesBase64Issue: failed attempt 1
java.lang.NoClassDefFoundError: org/apache/commons/net/util/Base64
at org.apache.kudu.backup.TableMetadata$.valueToString(TableMetadata.scala:351)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getTableMetadata$1(TableMetadata.scala:78)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getTableMetadata(TableMetadata.scala:61)
at org.apache.kudu.backup.KuduBackup$.doBackup(KuduBackup.scala:102)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$6(KuduBackup.scala:146)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$5(KuduBackup.scala:146)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:66)
at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1064)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1061)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:968)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:963)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.net.util.Base64
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 56 common frames omitted
During investigation I found another bug with BINARY range-partition
boundaries causing:
java.lang.ClassCastException: class java.nio.HeapByteBuffer cannot be
cast to class [B
This is fixed in the same valueToString switch; covered by
testBinaryPartitionBoundariesBase64Issue.
Running the repro test without the fix results in this stacktrace:
13:39:47.053 [ForkJoinPool-2-worker-1] ERROR org.apache.kudu.backup.KuduBackup$ -- Failed to back up table binary-partition-base64-issue
java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B
at org.apache.kudu.backup.TableMetadata$.valueToString(TableMetadata.scala:351)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getBoundValues$2(TableMetadata.scala:225)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getBoundValues(TableMetadata.scala:219)
at org.apache.kudu.backup.TableMetadata$.$anonfun$getRangePartitionMetadata$2(TableMetadata.scala:196)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.kudu.backup.TableMetadata$.getRangePartitionMetadata(TableMetadata.scala:195)
at org.apache.kudu.backup.TableMetadata$.getPartitionSchemaMetadata(TableMetadata.scala:125)
at org.apache.kudu.backup.TableMetadata$.getTableMetadata(TableMetadata.scala:106)
at org.apache.kudu.backup.KuduBackup$.doBackup(KuduBackup.scala:102)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$6(KuduBackup.scala:146)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.kudu.backup.KuduBackup$.$anonfun$run$5(KuduBackup.scala:146)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:66)
at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1064)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1061)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:389)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:968)
at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:963)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Before these fixes, running org.apache.kudu.backup.TestKuduBackup 100
times yielded 43 failures / 57 passes. After the fixes: 100/100 passes.
The random nature of test failures is because the problematic test:
testRandomBackupAndRestore uses randomized schema and data.
This is not a full RCA I wanted to deflake with reasonable solution;
I’ve opened KUDU-3694 to track root-cause follow-ups and additional
action items.
Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Reviewed-on: http://gerrit.cloudera.org:8080/23358
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Reviewed-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Gabriella Lotz <lo...@gmail.com>
Tested-by: Marton Greber <gr...@gmail.com>
(cherry picked from commit 724911497f7b2b2d6bc53f1c510ac0b82889bbb3)
Reviewed-on: http://gerrit.cloudera.org:8080/23433
Tested-by: Abhishek Chennaka <ac...@cloudera.com>
---
M java/kudu-backup-common/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
2 files changed, 153 insertions(+), 4 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved
Abhishek Chennaka: Verified
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has removed a vote on this change.
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR](branch-1.18.x) Deflake testRandomBackupAndRestore
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23433 )
Change subject: Deflake testRandomBackupAndRestore
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23433
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibbf05b23cb6c922ee880f4f1dda262be5e97452b
Gerrit-Change-Number: 23433
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Sep 2025 23:07:41 +0000
Gerrit-HasComments: No