You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bruce Robbins (Jira)" <ji...@apache.org> on 2022/04/21 22:28:00 UTC
[jira] [Created] (SPARK-38990) date_trunc and trunc both fail with format from column in inline table
Bruce Robbins created SPARK-38990:
-------------------------------------
Summary: date_trunc and trunc both fail with format from column in inline table
Key: SPARK-38990
URL: https://issues.apache.org/jira/browse/SPARK-38990
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1, 3.1.3, 3.3.0, 3.4.0
Reporter: Bruce Robbins
This fails:
{noformat}
create or replace temp view v1 as
select * from values ('week', timestamp'2012-01-01') as data(col1, col2);
select date_trunc(col1, col2) from v1;
{noformat}
It fails with a {{NullPointerException}}:
{noformat}
java.lang.NullPointerException: null
at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getAccessor$8(InternalRow.scala:141) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getAccessor$8$adapted(InternalRow.scala:141) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.BoundReference.eval(BoundAttribute.scala:40) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.TruncInstant.evalHelper(datetimeExpressions.scala:2117) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.TruncInstant.evalHelper$(datetimeExpressions.scala:2112) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.TruncTimestamp.evalHelper(datetimeExpressions.scala:2277) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.TruncTimestamp.eval(datetimeExpressions.scala:2295) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:157) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(InterpretedMutableProjection.scala:97) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$47.$anonfun$applyOrElse$77(Optimizer.scala:1987) ~[spark-catalyst_2.12-3
{noformat}
However, if you cache {{v1}}, then it works:
{noformat}
spark-sql> cache table v1;
Time taken: 2.086 seconds
spark-sql> select date_trunc(col1, col2) from v1;
2011-12-26 00:00:00
Time taken: 0.172 seconds, Fetched 1 row(s)
spark-sql>
{noformat}
Both {{date_trunc}} and {{trunc}} exhibit this behavior.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org