[jira] [Created] (SPARK-45926) The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly

["Asif (Jira)" <ji...@apache.org>]

Asif created SPARK-45926:
----------------------------

             Summary: The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly 
                 Key: SPARK-45926
                 URL: https://issues.apache.org/jira/browse/SPARK-45926
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.5.1
            Reporter: Asif


These   InMemoryV2FilterBatchScan and InMemoryBatchScan test classes are not implementing hashCode and equals correctly as they are not taking into account the pushed runtime filters. As a result they are unable to expose the TPCDS test issues which can show whether the reuse of exchange is happening correctly or not.
If these classes implement equals and hashCode taking into account the pushed runtime filters,  we would see that TPCDS Q14b which should ideally be reusing the exchange containing Union ,  is not happening due to multiple bugs which surface in AQE.

Actual V2 DataSources  like iceberg correctly implement equals and hashCode taking into account pushed runtime filters , which also expose the same issue of reuse of exchnage not happening



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org