You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bruce Robbins (Jira)" <ji...@apache.org> on 2022/01/30 23:02:00 UTC
[jira] [Commented] (SPARK-38075) Hive script transform with order by and limit will return fake rows
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484468#comment-17484468 ]
Bruce Robbins commented on SPARK-38075:
---------------------------------------
It's a small iterator issue. I will make a PR shortly.
> Hive script transform with order by and limit will return fake rows
> -------------------------------------------------------------------
>
> Key: SPARK-38075
> URL: https://issues.apache.org/jira/browse/SPARK-38075
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.2, 3.2.0, 3.3.0
> Reporter: Bruce Robbins
> Priority: Major
> Labels: correctness
>
> For example:
> {noformat}
> create or replace temp view t as
> select * from values
> (1),
> (2),
> (3)
> as t(a);
> select transform(a)
> USING 'cat' AS (a int)
> FROM t order by a limit 10;
> {noformat}
> This returns:
> {noformat}
> NULL
> NULL
> NULL
> 1
> 2
> 3
> {noformat}
> Without {{order by}} and {{limit}}, the query returns:
> {noformat}
> 1
> 2
> 3
> {noformat}
> Spark script transform does not have this issue. That is, if {{spark.sql.catalogImplementation=in-memory}}, Spark does not return fake rows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org