SparkDF writing null values when not to database

[Aram Morcecian <am...@gmail.com>]

Hi everyone, I'm facing something weird. After doing some transformations
to a SparkDF I print some rows and I see those perfectly, but when I write
that to the database (postgres) one of the transformed columns is written
with all null values in it.
I tried doing it with DynamicFrames (AWS Glue) and it works ok, but since I
need to overwrite the table that I'm writing on, I need to use SparkDF.

I'm attaching the code that I'm using to do this. Could someone help
me with this?

Thank you

Re: SparkDF writing null values when not to database

[Sean Owen <sr...@gmail.com>]

What is null, what is the type, does it make sense in postgres, etc. Need
more info.

On Wed, Sep 22, 2021 at 9:18 AM Aram Morcecian <am...@gmail.com> wrote:

> Hi everyone, I'm facing something weird. After doing some transformations
> to a SparkDF I print some rows and I see those perfectly, but when I write
> that to the database (postgres) one of the transformed columns is written
> with all null values in it.
> I tried doing it with DynamicFrames (AWS Glue) and it works ok, but since
> I need to overwrite the table that I'm writing on, I need to use SparkDF.
>
> I'm attaching the code that I'm using to do this. Could someone help
> me with this?
>
> Thank you
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org