You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/21 07:35:32 UTC
[GitHub] [incubator-pinot] khanhpv opened a new issue #6471: Spark-Pinot Connector regexp_like not worked
khanhpv opened a new issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471
Please make a confirmation about this. We have configured spark with
usePushDownFilters = true
but it seems spark not push down filters to pinot server, and exception thrown:
org.apache.spark.sql.AnalysisException: Undefined function: 'REGEXP_LIKE'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 44
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$11.$anonfun$applyOrElse$69(Analyzer.scala:1396)
at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:53)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$11.applyOrElse(Analyzer.scala:1395)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$11.applyOrElse(Analyzer.scala:1386)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:258)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:258)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:263)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:328)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:263)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:328)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:186)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:326)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:263)
...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765101585
Is there any chance to push the filters using regexp_like to pinot server instead of running filter on spark to optimize using resource? @mangrrua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765324592
I feel in this case, we can still write the query in SparkSQL `LIKE` syntax. Then in the filter Push Down code (https://github.com/apache/incubator-pinot/blob/master/pinot-connectors/pinot-spark-connector/src/main/scala/org/apache/pinot/connector/spark/connector/FilterPushDown.scala#L54), the connector can rewrite the query to Pinot `regexp_like` expression, then query Pinot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765461480
> I feel in this case, we can still write the query in SparkSQL `LIKE` syntax. Then in the filter Push Down code (https://github.com/apache/incubator-pinot/blob/master/pinot-connectors/pinot-spark-connector/src/main/scala/org/apache/pinot/connector/spark/connector/FilterPushDown.scala#L54), the connector can rewrite the query to Pinot `regexp_like` expression, then query Pinot.
I'll try this. Thank you so much, mangrrua and fx19880617
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-764577070
Yes, connector uses SQL instead of PQL. So it does not support regexp_like function. Use stringContains or some simirlar methods if it's fit your case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv edited a comment on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv edited a comment on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-766490730
It's currently not working, everywhen I execute statements with StringContains, the Pinot server returns LIKE enum not found. I have read the spark connector code, and used any filter push options there, but not luck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765101585
Is there any chance to push the filters using regexp_like to pinot server instead of running filter on spark to optimize using resource? @mangrrua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-766490730
It's currently not working, everywhen I execute statements with StringContains, the Pinot server returns LIKE enum not found.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-764479327
I think REGEXP_LIKE is pinot specific function, which is not registered in spark SQL. So you may need to register it then use it.
cc @mangrrua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-769948460
@fx19880617 Doesn't pinot sql support `like` keyword?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765462272
> I think it's not a good idea. Pinot will start using SQL instead of PQL completely. Also, connector uses SQL compiler of Pinot, and I don't know SQL compiler supports regexp_like function or not.
When this release?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-764479327
I think REGEXP_LIKE is pinot specific function, which is not registered in spark SQL. So you may need to register it then use it.
cc @mangrrua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv edited a comment on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv edited a comment on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-766490730
It's currently not working, everywhen I execute statements with StringContains, the Pinot server returns LIKE enum not found. I have read the spark connector code, and used any filter push options there, but not luck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] khanhpv commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
khanhpv commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-766490730
It's currently not working, everywhen I execute statements with StringContains, the Pinot server returns LIKE enum not found.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765611219
> > I think it's not a good idea. Pinot will start using SQL instead of PQL completely. Also, connector uses SQL compiler of Pinot, and I don't know SQL compiler supports regexp_like function or not.
>
> When this release?
we need to make a code change for this :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-766331143
Actually, methods below uses SQL `like` method.
```
case StringStartsWith(attr, value) => s"$attr LIKE '$value%'"
case StringEndsWith(attr, value) => s"$attr LIKE '%$value'"
case StringContains(attr, value) => s"$attr LIKE '%$value%'"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-764577070
Yes, connector uses SQL instead of PQL. So it does not support regexp_like function. Use stringContains or some simirlar methods if it's fit your case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765324592
I feel in this case, we need to write the query in SparkSQL syntax and in the filter PushDown code (https://github.com/apache/incubator-pinot/blob/master/pinot-connectors/pinot-spark-connector/src/main/scala/org/apache/pinot/connector/spark/connector/FilterPushDown.scala#L54) to rewrite the query to Pinot `regexp_like` expression, then query Pinot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on issue #6471: Spark-Pinot Connector regexp_like not worked
Posted by GitBox <gi...@apache.org>.
mangrrua commented on issue #6471:
URL: https://github.com/apache/incubator-pinot/issues/6471#issuecomment-765272112
I think it's not a good idea. Pinot will start using SQL instead of PQL completely. Also, connector uses SQL compiler of Pinot, and I don't know SQL compiler supports regexp_like function or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org