[GitHub] spark issue #16138: [SPARK-16609] Add to_date/to_timestamp with format funct...

[YannByron <gi...@git.apache.org>]

Github user YannByron commented on the issue:

    https://github.com/apache/spark/pull/16138
  
    @anabranch 
    
    I met a failure case when using `to_date`, like this:
    ```
     spark-sql> select func(to_date('2017-08-25 12:00:00'));
    ```
    `func` is a custom udf which extends `org.apache.hadoop.hive.ql.udf.generic.GenericUDF`.
    
    And this following exception happened:
    ```
    Error in query: No handler for Hive UDF 'xxx.xxx.func': java.lang.UnsupportedOperationException: Cannot evaluate expression: to_date(2017-08-25 12:00:00, None);
    ```
    
    I locate this problem. When resolving function `func(to_date('2017-08-25 12:00:00'))`(at this time Spark has already finished the resolving to `func`'s child `to_date('2017-08-25 12:00:00')`),  it will call `func`'s FunctionBuilder that is defined in [HiveSessionCatalog.makeFunctionBuilder](https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala)). 
    In it, the GenericUDF instance will be forced to check input data type. During this process, All of the child catalyst expression which are the parameters of the subclass of GenericUDF, need to be mapped to `ObjectInspector`(see details: [HiveInspector.toInspector](https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala)).  Inside, the child expression need to be evaluated though `eval()` firstly, if expression doesn't match with `Literal`.
    
    ```
    def toInspector(expr: Expression): ObjectInspector = expr match {
        case Literal(value, XXXType) =>
            //...
        // ideally, we don't test the foldable here(but in optimizer), however, some of the
        // Hive UDF / UDAF requires its argument to be constant objectinspector, we do it eagerly.
        case _ if expr.foldable => toInspector(Literal.create(expr.eval(), expr.dataType))
    }
    ```
    
    In my case, the child expression is `to_date('2017-08-25 12:00:00')` that is the instance of [ParseToDate](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala) (extends [RuntimeReplaceable](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala)). When calling its `eval()`,  we call the [Unevaluable](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala)'s `eval()` that will throw this exception mentioned above.
    ```
    trait Unevaluable extends Expression {
    
      final override def eval(input: InternalRow = null): Any =
        throw new UnsupportedOperationException(s"Cannot evaluate expression: $this")
    
      final override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode =
        throw new UnsupportedOperationException(s"Cannot evaluate expression: $this")
    }
    ```
    
    Considering that we barely use the second parameter `pattern`, I remap `to_date` to `ToDate` again to fix the problem.
    
    So, does my case relate to this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org