You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by bomeng <gi...@git.apache.org> on 2016/04/14 01:08:32 UTC

[GitHub] spark pull request: [SPARK-14541] [SQL] [WIP] SQL function: IFNULL...

GitHub user bomeng opened a pull request:

    https://github.com/apache/spark/pull/12373

    [SPARK-14541] [SQL] [WIP] SQL function: IFNULL, NULLIF, NVL and NVL2

    ## What changes were proposed in this pull request?
    I am trying to implement functions `NULLIF` in this PR. The meaning of NULLIF can be found here:
    [NULLIF( )](https://oracle-base.com/articles/misc/null-related-functions#nullif)
    
    ## How was this patch tested?
    Test cases were added.
    
    ## JIRA related
    [SPAKR-14541](https://issues.apache.org/jira/browse/SPARK-14541)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bomeng/spark SPARK-14541

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/12373.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12373
    
----
commit c479394e1a6aa1588544357a0aa76054cb813088
Author: bomeng <bm...@us.ibm.com>
Date:   2016-04-13T22:53:47Z

    support of NULLIF()

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] [WIP] SQL function: IFNULL...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12373#issuecomment-210128459
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55832/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12373#issuecomment-210648046
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55948/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by davies <gi...@git.apache.org>.

Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12373#discussion_r59813149
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
    @@ -1015,6 +1015,33 @@ object functions {
       def nanvl(col1: Column, col2: Column): Column = withExpr { NaNvl(col1.expr, col2.expr) }
     
       /**
    +   * Returns null if col1 equals to col2, otherwise returns col1.
    +   *
    +   * @group normal_funcs
    +   * @since 2.0.0
    +   */
    +  def nullif(col1: Column, col2: Column): Column = withExpr { NullIf(col1.expr, col2.expr)}
    +
    +  /**
    +   * Returns col2 if col1 equals is null, otherwise returns col1.
    +   *
    +   * @group normal_funcs
    +   * @since 2.0.0
    +   */
    +  def nvl(col1: Column, col2: Column): Column = withExpr { Nvl(col1.expr, col2.expr)}
    +  def ifnull(col1: Column, col2: Column): Column = withExpr { Nvl(col1.expr, col2.expr)}
    --- End diff --
    
    docs


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by bomeng <gi...@git.apache.org>.

Github user bomeng commented on the pull request:

    https://github.com/apache/spark/pull/12373#issuecomment-210300785
  
    I will address these issues tomorrow! Thank you all!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by rxin <gi...@git.apache.org>.

Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12373#discussion_r59813273
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala ---
    @@ -128,6 +128,143 @@ case class IsNaN(child: Expression) extends UnaryExpression
     }
     
     /**
    + * An Expression accepts two parameters and returns null if both parameters are equal.
    + * If they are not equal, the first parameter value is returned.
    + */
    +@ExpressionDescription(
    +  usage = "_FUNC_(a,b) - Returns null if a equals to b, or a otherwise.")
    +case class NullIf(left: Expression, right: Expression) extends BinaryExpression {
    +  override def nullable: Boolean = true
    +  override def dataType: DataType = left.dataType
    +
    +  override def eval(input: InternalRow): Any = {
    +    val valueLeft = left.eval(input)
    +    val valueRight = right.eval(input)
    +    if (valueLeft.equals(valueRight)) {
    +      null
    +    } else {
    +      valueLeft
    +    }
    +  }
    +
    +  override def genCode(ctx: CodegenContext, ev: ExprCode): String = {
    +    val leftGen = left.gen(ctx)
    +    val rightGen = right.gen(ctx)
    +    s"""
    +      ${leftGen.code}
    +      ${rightGen.code}
    +      boolean ${ev.isNull} = false;
    +      ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
    +      if (${ctx.genEqual(dataType, leftGen.value, rightGen.value)}) {
    +        ${ev.isNull} = true;
    +      } else {
    +        ${ev.value} = ${leftGen.value};
    +      }
    +    """
    +  }
    +}
    +
    +/**
    + * An Expression accepts two parameters and returns the second parameter if the value
    + * in the first parameter is null; if the first parameter is any value other than null,
    + * it is returned unchanged.
    + */
    +@ExpressionDescription(
    +  usage = "_FUNC_(a,b) - Returns b if a is null, or a otherwise.")
    +case class Nvl(left: Expression, right: Expression) extends BinaryExpression {
    +  override def nullable: Boolean = false
    +  override def dataType: DataType = left.dataType
    +
    +  override def eval(input: InternalRow): Any = {
    +    val valueLeft = left.eval(input)
    +    val valueRight = right.eval(input)
    +    if (valueLeft == null) {
    +      valueRight
    +    } else {
    +      valueLeft
    +    }
    +  }
    +
    +  override def genCode(ctx: CodegenContext, ev: ExprCode): String = {
    +    val leftGen = left.gen(ctx)
    +    val rightGen = right.gen(ctx)
    +    s"""
    +       ${leftGen.code}
    +       ${rightGen.code}
    +       boolean ${ev.isNull} = false;
    +       ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
    +     """ +
    +    {
    +      if (ctx.isPrimitiveType(dataType)) {
    +        s"""
    +          ${ev.value} = ${leftGen.value};
    +        """
    +      } else {
    +        s"""
    +          if (${leftGen.value} == null) {
    +            ${ev.value} = ${rightGen.value};
    +          } else {
    +            ${ev.value} = ${leftGen.value};
    +          }
    +        """
    +      }
    +    }
    +  }
    +}
    +
    +/**
    + * An Expression accepts three parameters and returns the second parameter if the first parameter
    + * value is not null; if the first parameter is null, it returns the third parameter.
    + */
    +@ExpressionDescription(
    +  usage = "_FUNC_(a,b) - Returns b if a is null, or a otherwise.")
    --- End diff --
    
    the example is wrong


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12373#issuecomment-210648042
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by rxin <gi...@git.apache.org>.

Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12373#discussion_r59813226
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala ---
    @@ -128,6 +128,143 @@ case class IsNaN(child: Expression) extends UnaryExpression
     }
     
     /**
    + * An Expression accepts two parameters and returns null if both parameters are equal.
    + * If they are not equal, the first parameter value is returned.
    + */
    +@ExpressionDescription(
    +  usage = "_FUNC_(a,b) - Returns null if a equals to b, or a otherwise.")
    +case class NullIf(left: Expression, right: Expression) extends BinaryExpression {
    +  override def nullable: Boolean = true
    +  override def dataType: DataType = left.dataType
    +
    +  override def eval(input: InternalRow): Any = {
    +    val valueLeft = left.eval(input)
    +    val valueRight = right.eval(input)
    +    if (valueLeft.equals(valueRight)) {
    +      null
    +    } else {
    +      valueLeft
    +    }
    +  }
    +
    +  override def genCode(ctx: CodegenContext, ev: ExprCode): String = {
    +    val leftGen = left.gen(ctx)
    +    val rightGen = right.gen(ctx)
    +    s"""
    +      ${leftGen.code}
    +      ${rightGen.code}
    +      boolean ${ev.isNull} = false;
    +      ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
    +      if (${ctx.genEqual(dataType, leftGen.value, rightGen.value)}) {
    +        ${ev.isNull} = true;
    +      } else {
    +        ${ev.value} = ${leftGen.value};
    +      }
    +    """
    +  }
    +}
    +
    +/**
    + * An Expression accepts two parameters and returns the second parameter if the value
    + * in the first parameter is null; if the first parameter is any value other than null,
    + * it is returned unchanged.
    + */
    +@ExpressionDescription(
    +  usage = "_FUNC_(a,b) - Returns b if a is null, or a otherwise.")
    +case class Nvl(left: Expression, right: Expression) extends BinaryExpression {
    --- End diff --
    
    isn't this just coalesce?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12373#issuecomment-210605685
  
    **[Test build #55947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55947/consoleFull)** for PR 12373 at commit [`451e5fb`](https://github.com/apache/spark/commit/451e5fbec0a4fb145471b87ccc7d9578a07b267a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-14541] [SQL] SQL function: IFNULL, NULL...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/12373


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org