You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Abhijeet Kumar <ab...@sentienz.com> on 2018/06/27 17:45:13 UTC

Not able to overwrite cassandra table using Spark

Hello Team,

I’m creating a dataframe out of cassandra table using datastax spark connector. After making some modification into the dataframe, I’m trying to put that dataframe back to the Cassandra table by overwriting the old content. For that the piece of code is:

modifiedList.write.format("org.apache.spark.sql.cassandra")
      .options(Map( "table" -> "list", "keyspace" -> "journaling", "confirm.truncate" -> "true"))
      .mode(Overwrite).save

It’s not showing any error and seems like working fine, but when I’m checking the Cassandra table back, there is no content inside it. Everything is deleted. I’m really worried about this behaviour because this may delete some useful content (I’m sure about overwriting the content and fully understand the consequences).

Thanks,
Abhijeet Kumar

Re: Not able to overwrite cassandra table using Spark

Posted by Siva Samraj <sa...@gmail.com>.

You can try with this, it will work

val finaldf = merchantdf.write.

      format("org.apache.spark.sql.cassandra")

      .mode(SaveMode.Overwrite)

      .option("confirm.truncate", true)

      .options(Map("table" -> "tablename", "keyspace" -> "keyspace"))

      .save()


On Wed 27 Jun, 2018, 11:15 PM Abhijeet Kumar, <ab...@sentienz.com>
wrote:

> Hello Team,
>
> I’m creating a dataframe out of cassandra table using datastax spark
> connector. After making some modification into the dataframe, I’m trying to
> put that dataframe back to the Cassandra table by overwriting the old
> content. For that the piece of code is:
>
> modifiedList.write.format("org.apache.spark.sql.cassandra")
>       .options(Map( "table" -> "list", "keyspace" -> "journaling",
> "confirm.truncate" -> "true"))
>       .mode(Overwrite).save
>
> It’s not showing any error and seems like working fine, but when I’m
> checking the Cassandra table back, there is no content inside it.
> Everything is deleted. I’m really worried about this behaviour because this
> may delete some useful content (I’m sure about overwriting the content and
> fully understand the consequences).
>
> Thanks,
> Abhijeet Kumar
>