You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "murrelljenna (via GitHub)" <gi...@apache.org> on 2024/03/20 20:33:55 UTC
[PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
murrelljenna opened a new pull request, #45619:
URL: https://github.com/apache/spark/pull/45619
This PR is a rebased and updated version of a [previous pr](https://github.com/apache/spark/pull/24801) by @etspaceman from 2019. This PR allows the spark kinesis library to work with localstack instances of kinesis.
I have not added any changes to @etspaceman 's original code, except to remove the `cloudwatchMetricsLevel` value in the original PR, as subsequent PRs have added that same value in since.
### What changes were proposed in this pull request?
Currently the spark kinesis library allows you to override the kinesis endpoint, but not the dynamodb endpoint the kinesis consumer will use to manage leases. This PR adds that configurable url into the kinesis input stream builder.
### Why are the changes needed?
Localstack has only become more popular for integration testing since the original PR. Currently, one cannot use spark streaming and localstack kinesis together, as the kinesis consumer will only attempt to manage leases through a remote aws dynamo instance.
### Does this PR introduce _any_ user-facing change?
This PR will allow the user to specify their own dynamo endpoint, much like how they can currently specify their own kinesis endpoint.
### How was this patch tested?
This PR adds unit tests to cover this new value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on code in PR #45619:
URL: https://github.com/apache/spark/pull/45619#discussion_r1534149417
##########
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala:
##########
@@ -169,13 +172,19 @@ private[kinesis] class KinesisReceiver[T](
.withMetricsLevel(metricsLevel)
.withMetricsEnabledDimensions(metricsEnabledDimensions.asJava)
+ val withOptionalConfiguration = dynamoDBEndpointUrl match {
+ case Some(endpoint) => baseClientLibConfiguration
+ .withDynamoDBEndpoint(endpoint).withMetricsLevel(metricsLevel)
Review Comment:
Yea, good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2013947923
Just as a heads up, I'll be unavailable tomorrow and monday, but will be back on this pr come tuesday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020829179
Ya, unfortunately, that could be one of the root cause why Apache Spark community was unable to approve and accept this domain like the previous PR.
https://github.com/apache/spark/pull/24801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2010655319
Could you set `ENABLE_KINESIS_TESTS=1` and run the tests in your environment? It would be great if you can paste your result into the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2026089400
Do I need to set up my own aws environment to run these tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2014048519
Ya, let's continue next week because the above result looks insufficient to me.
I expected the following with `ENABLE_KINESIS_TESTS=1`.
```
$ build/sbt "streaming-kinesis-asl/test" -Pkinesis-asl
...
[info] KinesisInputDStreamBuilderSuite:
[info] - should raise an exception if the StreamingContext is missing (12 milliseconds)
[info] - should raise an exception if the stream name is missing (3 milliseconds)
[info] - should raise an exception if the checkpoint app name is missing (1 millisecond)
[info] - should propagate required values to KinesisInputDStream (71 milliseconds)
[info] - should propagate default values to KinesisInputDStream (2 milliseconds)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appendedaming-kinesis-asl / Test / executeTests 0s
[info] - should propagate custom non-auth values to KinesisInputDStream (481 milliseconds)
[info] - old Api should throw UnsupportedOperationExceptionexception with AT_TIMESTAMP (2 milliseconds)
[info] WithoutAggregationKinesisBackedBlockRDDSuite:
[info] - Basic reading from Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available in both block manager and Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available only in block manager, not in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available only in Kinesis, not in block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available partially in block manager, rest in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Test isBlockValid skips block fetching from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Test whether RDD is valid after removing blocks from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] KinesisReceiverSuite:
[info] - process records including store and set checkpointer (25 milliseconds)
[info] - split into multiple processes if a limitation is set (2 milliseconds)
[info] - shouldn't store and update checkpointer when receiver is stopped (2 milliseconds)
[info] - shouldn't update checkpointer when exception occurs during store (4 milliseconds)
[info] - shutdown should checkpoint if the reason is TERMINATE (3 milliseconds)
[info] - shutdown should not checkpoint if the reason is something other than TERMINATE (2 milliseconds)
[info] - retry success on first attempt (2 milliseconds)
[info] - retry success on second attempt after a Kinesis throttling exception (40 milliseconds)
[info] - retry success on second attempt after a Kinesis dependency exception (49 milliseconds)
[info] - retry failed after a shutdown exception (3 milliseconds)
[info] - retry failed after an invalid state exception (2 milliseconds)
[info] - retry failed after unexpected exception (2 milliseconds)
[info] - retry failed after exhausting all retries (108 milliseconds)
[info] WithAggregationKinesisBackedBlockRDDSuite:
[info] - Basic reading from Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available in both block manager and Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available only in block manager, not in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available only in Kinesis, not in block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Read data available partially in block manager, rest in Kinesis [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Test isBlockValid skips block fetching from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Test whether RDD is valid after removing blocks from block manager [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] WithAggregationKinesisStreamSuite:
[info] - RDD generation (16 milliseconds)
[info] - basic operation [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - custom message handling [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Kinesis read with custom configurations (2 milliseconds)
[info] - split and merge shards in a stream [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - failure recovery [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Prepare KinesisTestUtils [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] KinesisCheckpointerSuite:
[info] - checkpoint is not called twice for the same sequence number (3 milliseconds)
[info] - checkpoint is called after sequence number increases (1 millisecond)
[info] - should checkpoint if we have exceeded the checkpoint interval (2 milliseconds)
[info] - shouldn't checkpoint if we have not exceeded the checkpoint interval (0 milliseconds)
[info] - should not checkpoint for the same sequence number (1 millisecond)
[info] - removing checkpointer checkpoints one last time (1 millisecond)
[info] - if checkpointing is going on, wait until finished before removing and checkpointing (65 milliseconds)
[info] SparkAWSCredentialsBuilderSuite:
[info] - should build DefaultCredentials when given no params (1 millisecond)
[info] - should build BasicCredentials (0 milliseconds)
[info] - should build STSCredentials (2 milliseconds)
[info] - SparkAWSCredentials classes should be serializable (4 milliseconds)
[info] WithoutAggregationKinesisStreamSuite:
[info] - RDD generation (1 millisecond)
[info] - basic operation [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - custom message handling [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Kinesis read with custom configurations (1 millisecond)
[info] - split and merge shards in a stream [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - failure recovery [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] - Prepare KinesisTestUtils [enable by setting env var ENABLE_KINESIS_TESTS=1] !!! IGNORED !!!
[info] Test run started (JUnit Jupiter)
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilderOldApi() started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilder() started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.133s
[info] ScalaTest
[info] Run completed in 2 seconds, 625 milliseconds.
[info] Total number of tests run: 35
[info] Suites: completed 8, aborted 0
[info] Tests: succeeded 35, failed 0, canceled 0, ignored 24, pending 0
[info] All tests passed.
[info] Passed: Total 37, Failed 0, Errors 0, Passed 37, Ignored 24
[success] Total time: 39 s, completed Mar 21, 2024, 4:34:09 PM
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020819428
Thanks! That was way faster.
This is likely some local laptop issue I'm having, but I'm getting class not found errors. Also, I have not set up to talk to any particular aws resources on this laptop for spark, so I'm not sure how these tests will work.
```
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
WARNING: Using incubator modules: jdk.incubator.vector, jdk.incubator.foreign
[info] KinesisInputDStreamBuilderSuite:
[info] - should raise an exception if the StreamingContext is missing (25 milliseconds)
[info] - should raise an exception if the stream name is missing (3 milliseconds)
[info] - should raise an exception if the checkpoint app name is missing (1 millisecond)
[info] - should propagate required values to KinesisInputDStream (81 milliseconds)
[info] - should propagate default values to KinesisInputDStream (2 milliseconds)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[info] - should propagate custom non-auth values to KinesisInputDStream (511 milliseconds)
[info] - old Api should throw UnsupportedOperationExceptionexception with AT_TIMESTAMP (3 milliseconds)
Kinesis tests that actually send data has been enabled by setting the environment
variable ENABLE_KINESIS_TESTS to 1. This will create Kinesis Streams and
DynamoDB tables in AWS. Please be aware that this may incur some AWS costs.
By default, the tests use the endpoint URL https://kinesis.us-west-2.amazonaws.com to create Kinesis streams.
To change this endpoint URL to a different region, you can set the environment variable
KINESIS_TEST_ENDPOINT_URL to the desired endpoint URL
(e.g. KINESIS_TEST_ENDPOINT_URL="https://kinesis.us-west-2.amazonaws.com").
Using endpoint URL https://kinesis.us-west-2.amazonaws.com for creating Kinesis streams for tests.
[info] WithoutAggregationKinesisBackedBlockRDDSuite:
[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite *** ABORTED *** (1 second, 93 milliseconds)
[info] java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.PropertyNamingStrategy$PascalCaseStrategy
[info] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[info] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[info] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.PropertyNamingStrategy$PascalCaseStrategy
[error] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[error] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[error] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] KinesisReceiverSuite:
[info] - process records including store and set checkpointer (47 milliseconds)
[info] - split into multiple processes if a limitation is set (4 milliseconds)
[info] - shouldn't store and update checkpointer when receiver is stopped (5 milliseconds)
[info] - shouldn't update checkpointer when exception occurs during store (9 milliseconds)
[info] - shutdown should checkpoint if the reason is TERMINATE (7 milliseconds)
[info] - shutdown should not checkpoint if the reason is something other than TERMINATE (3 milliseconds)
[info] - retry success on first attempt (2 milliseconds)
[info] - retry success on second attempt after a Kinesis throttling exception (102 milliseconds)
[info] - retry success on second attempt after a Kinesis dependency exception (16 milliseconds)
[info] - retry failed after a shutdown exception (3 milliseconds)
[info] - retry failed after an invalid state exception (3 milliseconds)
[info] - retry failed after unexpected exception (4 milliseconds)
[info] - retry failed after exhausting all retries (61 milliseconds)
[info] WithAggregationKinesisBackedBlockRDDSuite:
[info] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite *** ABORTED *** (1 second, 12 milliseconds)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] WithAggregationKinesisStreamSuite:
[info] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite *** ABORTED *** (1 second, 5 milliseconds)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[error] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] KinesisCheckpointerSuite:
[info] - checkpoint is not called twice for the same sequence number (8 milliseconds)
[info] - checkpoint is called after sequence number increases (3 milliseconds)
[info] - should checkpoint if we have exceeded the checkpoint interval (4 milliseconds)
[info] - shouldn't checkpoint if we have not exceeded the checkpoint interval (2 milliseconds)
[info] - should not checkpoint for the same sequence number (3 milliseconds)
[info] - removing checkpointer checkpoints one last time (2 milliseconds)
[info] - if checkpointing is going on, wait until finished before removing and checkpointing (60 milliseconds)
[info] SparkAWSCredentialsBuilderSuite:
[info] - should build DefaultCredentials when given no params (3 milliseconds)
[info] - should build BasicCredentials (1 millisecond)
[info] - should build STSCredentials (1 millisecond)
[info] - SparkAWSCredentials classes should be serializable (4 milliseconds)
[info] WithoutAggregationKinesisStreamSuite:
[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 1 millisecond)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[error] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Test run started (JUnit Jupiter)
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilderOldApi() started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilder() started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.236s
[info] ScalaTest
[info] Run completed in 7 seconds, 455 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 16 s, completed Mar. 26, 2024, 11:48:50 a.m.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020793922
@murrelljenna Could you try the following command? (https://github.com/apache/spark/pull/45619#issuecomment-2014048519)
> $ build/sbt "streaming-kinesis-asl/test" -Pkinesis-asl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2037752508
@dongjoon-hyun , do these tests generally require one to configure their own aws kinesis environment? Have you run these tests before? This change is pretty simple, it'd be a shame if the pr were to die again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2013672115
So I said I'd paste shortly - that test run took 3 hours. Not so short after all. It seems I have no failed tests but some suites didn't run. I can post the whole output but that would be enormous.
```[info] Total number of tests run: 15254
[info] Suites: completed 621, aborted 0
[info] Tests: succeeded 15254, failed 0, canceled 46, ignored 96, pending 0
[info] All tests passed.
[info] Passed: Total 15398, Failed 0, Errors 0, Passed 15398, Ignored 96, Canceled 46
[error] (core / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (hive / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (mllib / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 11307 s (03:08:27), completed Mar. 21, 2024, 4:03:44 p.m.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2012483051
Thank you, @murrelljenna !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020904893
Ah, I see, that may be a complicating factor. Here is my output on master.
```
[info] KinesisInputDStreamBuilderSuite:
[info] - should raise an exception if the StreamingContext is missing (15 milliseconds)
[info] - should raise an exception if the stream name is missing (3 milliseconds)
[info] - should raise an exception if the checkpoint app name is missing (1 millisecond)
[info] - should propagate required values to KinesisInputDStream (79 milliseconds)
[info] - should propagate default values to KinesisInputDStream (2 milliseconds)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[info] - should propagate custom non-auth values to KinesisInputDStream (517 milliseconds)
[info] - old Api should throw UnsupportedOperationExceptionexception with AT_TIMESTAMP (4 milliseconds)
Kinesis tests that actually send data has been enabled by setting the environment
variable ENABLE_KINESIS_TESTS to 1. This will create Kinesis Streams and
DynamoDB tables in AWS. Please be aware that this may incur some AWS costs.
By default, the tests use the endpoint URL https://kinesis.us-west-2.amazonaws.com to create Kinesis streams.
To change this endpoint URL to a different region, you can set the environment variable
KINESIS_TEST_ENDPOINT_URL to the desired endpoint URL
(e.g. KINESIS_TEST_ENDPOINT_URL="https://kinesis.us-west-2.amazonaws.com").
Using endpoint URL https://kinesis.us-west-2.amazonaws.com for creating Kinesis streams for tests.
[info] WithoutAggregationKinesisBackedBlockRDDSuite:
[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite *** ABORTED *** (1 second, 108 milliseconds)
[info] java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.PropertyNamingStrategy$PascalCaseStrategy
[info] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[info] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[info] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.PropertyNamingStrategy$PascalCaseStrategy
[error] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[error] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[error] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] KinesisReceiverSuite:
[info] - process records including store and set checkpointer (43 milliseconds)
[info] - split into multiple processes if a limitation is set (4 milliseconds)
[info] - shouldn't store and update checkpointer when receiver is stopped (5 milliseconds)
[info] - shouldn't update checkpointer when exception occurs during store (8 milliseconds)
[info] - shutdown should checkpoint if the reason is TERMINATE (6 milliseconds)
[info] - shutdown should not checkpoint if the reason is something other than TERMINATE (3 milliseconds)
[info] - retry success on first attempt (4 milliseconds)
[info] - retry success on second attempt after a Kinesis throttling exception (95 milliseconds)
[info] - retry success on second attempt after a Kinesis dependency exception (63 milliseconds)
[info] - retry failed after a shutdown exception (3 milliseconds)
[info] - retry failed after an invalid state exception (3 milliseconds)
[info] - retry failed after unexpected exception (3 milliseconds)
[info] - retry failed after exhausting all retries (92 milliseconds)
[info] WithAggregationKinesisBackedBlockRDDSuite:
[info] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite *** ABORTED *** (1 second, 12 milliseconds)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] WithAggregationKinesisStreamSuite:
[info] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite *** ABORTED *** (1 second, 5 milliseconds)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[error] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] KinesisCheckpointerSuite:
[info] - checkpoint is not called twice for the same sequence number (6 milliseconds)
[info] - checkpoint is called after sequence number increases (3 milliseconds)
[info] - should checkpoint if we have exceeded the checkpoint interval (4 milliseconds)
[info] - shouldn't checkpoint if we have not exceeded the checkpoint interval (1 millisecond)
[info] - should not checkpoint for the same sequence number (2 milliseconds)
[info] - removing checkpointer checkpoints one last time (3 milliseconds)
[info] - if checkpointing is going on, wait until finished before removing and checkpointing (65 milliseconds)
[info] SparkAWSCredentialsBuilderSuite:
[info] - should build DefaultCredentials when given no params (2 milliseconds)
[info] - should build BasicCredentials (1 millisecond)
[info] - should build STSCredentials (1 millisecond)
[info] - SparkAWSCredentials classes should be serializable (3 milliseconds)
[info] WithoutAggregationKinesisStreamSuite:
[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 3 milliseconds)
[info] java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Cause: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[info] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[info] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Uncaught exception when running org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.services.kinesis.AmazonKinesisClient
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[error] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[error] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/PropertyNamingStrategy$PascalCaseStrategy [in thread "pool-1-thread-1"]
[error] at com.amazonaws.services.kinesis.AmazonKinesisClient.<clinit>(AmazonKinesisClient.java:86)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[error] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.$anonfun$beforeAll$1(KinesisBackedBlockRDDSuite.scala:45)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[error] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:26)
[error] at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:43)
[error] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[error] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[error] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Test run started (JUnit Jupiter)
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilderOldApi() started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite#testJavaKinesisDStreamBuilder() started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.24s
[info] ScalaTest
[info] Run completed in 7 seconds, 258 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 19 s, completed Mar. 26, 2024, 12:23:45 p.m.
```
It looks pretty much like the same output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020907178
We're using java 17, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2012474922
On it now, will paste shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2022751142
This error occurs here:
```
val client = new AmazonKinesisClient(KinesisTestUtils.getAWSCredentials())
```
Am I expected to have credentials to my own aws in order to run this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020790396
Weird, is there a way to just run the kinesis tests? Even with ENABLE_KINESIS_TESTS=1, dev/run-tests still spends 3 hours running every single suite of tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2020833781
BTW, could you give me some more data points, please, @murrelljenna ?
What is the result when you run the test on the vanilla `master`, `branch-3.5`, `branch-3.4`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2021870150
> Spark is using java 17, correct?
- `master` branch uses Java 17.
https://github.com/apache/spark/blob/7d87a94dd77f43120701e48a371324a4f5f2064b/pom.xml#L116
- `branch-3.5` and `branch-3.4` uses Java 8.
https://github.com/apache/spark/blob/8bcbf7701388a2da06369ae9317d7707624edba0/pom.xml#L115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAMS][Kinesis] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45619:
URL: https://github.com/apache/spark/pull/45619#discussion_r1534064717
##########
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala:
##########
@@ -169,13 +172,19 @@ private[kinesis] class KinesisReceiver[T](
.withMetricsLevel(metricsLevel)
.withMetricsEnabledDimensions(metricsEnabledDimensions.asJava)
+ val withOptionalConfiguration = dynamoDBEndpointUrl match {
+ case Some(endpoint) => baseClientLibConfiguration
+ .withDynamoDBEndpoint(endpoint).withMetricsLevel(metricsLevel)
Review Comment:
nit. Shall we remove `.withMetricsLevel(metricsLevel)` like the following because we already have it in the above?
```scala
- .withDynamoDBEndpoint(endpoint).withMetricsLevel(metricsLevel)
+ .withDynamoDBEndpoint(endpoint)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "murrelljenna (via GitHub)" <gi...@apache.org>.
murrelljenna commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2037794818
I see, well in theory the kinesis tests could use localstack kinesis (ironically its this PR that would make that possible), but that would be a bit of an effort.
I feel like this pr is at an impasse then, no? Are there any community members that have the infrastructure to run these tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-27950][DSTREAM] Configurable dynamodb url so kinesis-asl works with localstack [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45619:
URL: https://github.com/apache/spark/pull/45619#issuecomment-2037758698
> @dongjoon-hyun , do these tests generally require one to configure their own aws kinesis environment? Have you run these tests before? This change is pretty simple, it'd be a shame if the pr were to die again.
- https://github.com/apache/spark/pull/24801#issuecomment-2004417023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org