You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sebastian Piu <se...@gmail.com> on 2016/02/15 20:17:46 UTC
Check if column exists in Schema
Is there any way of checking if a given column exists in a Dataframe?
Re: Check if column exists in Schema
Posted by Sebastian Piu <se...@gmail.com>.
I'm trying to do it before the column is bound to a dataframe. I guess I'm
looking for something like col("x").isNull but that would return whether a
column exists or not
On Mon, Feb 15, 2016 at 10:31 PM Mohammed Guller <mo...@glassbeam.com>
wrote:
> The DataFrame class has a method named columns, which returns all column
> names as an array. You can then use the contains method in the Scala Array
> class to check whether a column exists.
>
>
>
> Mohammed
>
> Author: Big Data Analytics with Spark
> <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
>
>
>
> *From:* Sebastian Piu [mailto:sebastian.piu@gmail.com]
> *Sent:* Monday, February 15, 2016 11:21 AM
> *To:* user
> *Subject:* Re: Check if column exists in Schema
>
>
>
> I just realised this is a bit vague, I'm looking to create a function that
> looks into different columns to get a value. So depending on a type I might
> look into a given path or another (which might or might not exist).
>
>
>
> Example if column *some.path.to.my.date *exists I'd return that, if it
> doesn't or it is null, i'd get it from some other place
>
>
>
> On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piu <se...@gmail.com>
> wrote:
>
> Is there any way of checking if a given column exists in a Dataframe?
>
>
RE: Check if column exists in Schema
Posted by Mohammed Guller <mo...@glassbeam.com>.
The DataFrame class has a method named columns, which returns all column names as an array. You can then use the contains method in the Scala Array class to check whether a column exists.
Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Sebastian Piu [mailto:sebastian.piu@gmail.com]
Sent: Monday, February 15, 2016 11:21 AM
To: user
Subject: Re: Check if column exists in Schema
I just realised this is a bit vague, I'm looking to create a function that looks into different columns to get a value. So depending on a type I might look into a given path or another (which might or might not exist).
Example if column some.path.to.my.date exists I'd return that, if it doesn't or it is null, i'd get it from some other place
On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piu <se...@gmail.com>> wrote:
Is there any way of checking if a given column exists in a Dataframe?
Re: Check if column exists in Schema
Posted by Sebastian Piu <se...@gmail.com>.
I just realised this is a bit vague, I'm looking to create a function that
looks into different columns to get a value. So depending on a type I might
look into a given path or another (which might or might not exist).
Example if column *some.path.to.my.date *exists I'd return that, if it
doesn't or it is null, i'd get it from some other place
On Mon, Feb 15, 2016 at 7:17 PM Sebastian Piu <se...@gmail.com>
wrote:
> Is there any way of checking if a given column exists in a Dataframe?
>