You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sumeet Sharma (Jira)" <ji...@apache.org> on 2021/03/19 16:16:00 UTC
[jira] [Updated] (SPARK-34804) registerFunction shouldnt be logging
warning message for same function being re-registered
[ https://issues.apache.org/jira/browse/SPARK-34804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sumeet Sharma updated SPARK-34804:
----------------------------------
Description:
{code:java}
test("function registry warning") {
implicit val ss = spark
import ss.implicits._
val dd = Seq((1),(2)).toDF("a")
val dd1 = udf((i: Int) => i*2)
(1 to 4).foreach {
_ =>
dd.sparkSession.udf.register("function", dd1)
Thread.sleep(1000)
}
dd.withColumn("aa", expr("function(a)")).show(10)
}{code}
logs:
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; the static sql configurations will not take effect.
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
21/03/19 22:39:43 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
21/03/19 22:39:44 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
21/03/19 22:39:45 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
+----+--+
|a|aa|
+----+--+
|1|2|
|2|4|
+----+--+
Basically in the FunctionRegistry implementation
{code:java}
override def registerFunction(
name: FunctionIdentifier,
info: ExpressionInfo,
builder: FunctionBuilder): Unit = synchronized {
val normalizedName = normalizeFuncName(name)
val newFunction = (info, builder)
functionBuilders.put(normalizedName, newFunction) match {
case Some(previousFunction) if previousFunction != newFunction =>
logWarning(s"The function $normalizedName replaced a previously registered function.")
case _ =>
}
}{code}
The *previousFunction != newFunction* equality comparison is incorrect
was:
{code:java}
test("function registry warning") {
implicit val ss = spark
import ss.implicits._
val dd = Seq((1),(2)).toDF("a")
val dd1 = udf((i: Int) => i*2)
(1 to 4).foreach {
_ =>
dd.sparkSession.udf.register("function", dd1)
Thread.sleep(1000)
}
dd.withColumn("aa", expr("function(a)")).show(10)
}{code}
logs:
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; the static sql configurations will not take effect.
21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
21/03/19 22:39:43 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
21/03/19 22:39:44 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
21/03/19 22:39:45 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
+---+---+
| a| aa|
+---+---+
| 1| 2|
| 2| 4|
+---+---+
> registerFunction shouldnt be logging warning message for same function being re-registered
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-34804
> URL: https://issues.apache.org/jira/browse/SPARK-34804
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Sumeet Sharma
> Priority: Minor
>
> {code:java}
> test("function registry warning") {
> implicit val ss = spark
> import ss.implicits._
> val dd = Seq((1),(2)).toDF("a")
> val dd1 = udf((i: Int) => i*2)
> (1 to 4).foreach {
> _ =>
> dd.sparkSession.udf.register("function", dd1)
> Thread.sleep(1000)
> }
> dd.withColumn("aa", expr("function(a)")).show(10)
> }{code}
> logs:
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; the static sql configurations will not take effect.
> 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; some spark core configurations may not take effect.
> 21/03/19 22:39:43 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
> 21/03/19 22:39:44 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
> 21/03/19 22:39:45 WARN SimpleFunctionRegistry : The function function replaced a previously registered function.
> +----+--+
> |a|aa|
> +----+--+
> |1|2|
> |2|4|
> +----+--+
> Basically in the FunctionRegistry implementation
> {code:java}
> override def registerFunction(
> name: FunctionIdentifier,
> info: ExpressionInfo,
> builder: FunctionBuilder): Unit = synchronized {
> val normalizedName = normalizeFuncName(name)
> val newFunction = (info, builder)
> functionBuilders.put(normalizedName, newFunction) match {
> case Some(previousFunction) if previousFunction != newFunction =>
> logWarning(s"The function $normalizedName replaced a previously registered function.")
> case _ =>
> }
> }{code}
> The *previousFunction != newFunction* equality comparison is incorrect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org