You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2016/02/13 03:19:13 UTC
org.apache.spark.sql.AnalysisException: undefined function lit;
I am trying to add a column with a constant value to my data frame. Any idea
what I am doing wrong?
Kind regards
Andy
DataFrame result =
String exprStr = "lit(" + time.milliseconds()+ ") as ms";
logger.warn("AEDWIP expr: {}", exprStr);
result.selectExpr("*", exprStr).show(false);
WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1 call
line:96 AEDWIP expr: lit(1455329175000) as ms
ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error
running job streaming job 1455329175000 ms.0
org.apache.spark.sql.AnalysisException: undefined function lit;
Re: org.apache.spark.sql.AnalysisException: undefined function lit;
Posted by Michael Armbrust <mi...@databricks.com>.
selectExpr just uses the SQL parser to interpret the string you give it.
So to get a string literal you would use quotes:
df.selectExpr("*", "'" + time.miliseconds() + "' AS ms")
On Fri, Feb 12, 2016 at 6:19 PM, Andy Davidson <
Andy@santacruzintegration.com> wrote:
> I am trying to add a column with a constant value to my data frame. Any
> idea what I am doing wrong?
>
> Kind regards
>
> Andy
>
>
> DataFrame result = …
>
> String exprStr = "lit(" + time.milliseconds()+ ") as ms";
>
> logger.warn("AEDWIP expr: {}", exprStr);
>
> result.selectExpr("*", exprStr).show(false);
>
> WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1
> call line:96 AEDWIP expr: lit(1455329175000) as ms
>
> ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error
> running job streaming job 1455329175000 ms.0
>
> org.apache.spark.sql.AnalysisException: undefined function lit;
>
>
>
>
Re: org.apache.spark.sql.AnalysisException: undefined function lit;
Posted by Sebastian Piu <se...@gmail.com>.
I've never done it that way but you can simply use the withColumn method in
data frames to do it.
On 13 Feb 2016 2:19 a.m., "Andy Davidson" <An...@santacruzintegration.com>
wrote:
> I am trying to add a column with a constant value to my data frame. Any
> idea what I am doing wrong?
>
> Kind regards
>
> Andy
>
>
> DataFrame result = …
>
> String exprStr = "lit(" + time.milliseconds()+ ") as ms";
>
> logger.warn("AEDWIP expr: {}", exprStr);
>
> result.selectExpr("*", exprStr).show(false);
>
> WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1
> call line:96 AEDWIP expr: lit(1455329175000) as ms
>
> ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error
> running job streaming job 1455329175000 ms.0
>
> org.apache.spark.sql.AnalysisException: undefined function lit;
>
>
>
>