Exception during writing a spark Dataframe to Redshift

[Sandeep Patra <sp...@gmail.com>]

This is where the exception occurs:

myAppDes.coalesce(1)
        .write
        .format("com.databricks.spark.redshift")
        .option("url", redshiftURL)
        .option("dbtable", redshiftTableName)
        .option("forward_spark_s3_credentials", "true")
        .option("tempdir", "s3a://zest-hevo-datalake/temp/data")
        .mode(SaveMode.Append)
        .save()

I have attached the stack trace with the email.

My build.sbt looks like:
version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "2.3.0"
val hadoopVersion = "3.1.2"

resolvers ++= Seq(
  "apache-snapshots" at "http://repository.apache.org/snapshots/",
  "redshift" at "https://s3.amazonaws.com/redshift-maven-repository/release
",
  "redshift" at "
http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release
",
  "jitpack" at "https://jitpack.io"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.hadoop" % "hadoop-aws" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-common" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion,
  "com.amazon.redshift" % "redshift-jdbc42-no-awssdk" % "1.2.15.1025",
  "com.databricks" %% "spark-redshift" % "3.0.0-preview1",
  "com.amazon.redshift" % "redshift-jdbc42" % "1.2.1.1001"
)

dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" %
"2.8.8"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" %
"2.8.8"
dependencyOverrides += "com.fasterxml.jackson.module" %
"jackson-module-scala_2.11" % "2.8.8"

I have checked and I can read data from the same Redshift Instance. I also
have created table in redshift. (I also tried it without creating table but
got the same error)