You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "tsarangi (Jira)" <ji...@apache.org> on 2022/01/04 20:41:00 UTC
[jira] [Updated] (SPARK-37811) Broadcast Join throws HintErrorLogger for joins with multiple tables
[ https://issues.apache.org/jira/browse/SPARK-37811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tsarangi updated SPARK-37811:
-----------------------------
Description:
Following query throws HintLoggerError Warnings in v3.1.2.
{code:java}
// code placeholder
val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " +
" L1.v1 AS L1V1 " +
" L4.* " +
" FROM L1 " +
" INNER JOIN L2 ON L2.id = L1.id " +
" INNER JOIN L3 ON L3.id = L1.id " +
" LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 "
{code}
Following is the warning it thorws during runtime:
{code:java}
WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code}
The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4).
I have set \{{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3.
Let me know if you need more info.
was:
Following query throws HintLoggerError Warnings in v3.1.2.
{code:java}
// code placeholder
val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " +
" FROM L1 " +
" INNER JOIN L2 ON L2.id = L1.id " +
" INNER JOIN L3 ON L3.id = L1.id " +
" LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 "
{code}
Following is the warning it thorws during runtime:
{code:java}
WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code}
The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4).
I have set {{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3.
Let me know if you need more info.
> Broadcast Join throws HintErrorLogger for joins with multiple tables
> --------------------------------------------------------------------
>
> Key: SPARK-37811
> URL: https://issues.apache.org/jira/browse/SPARK-37811
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.2
> Reporter: tsarangi
> Priority: Minor
> Labels: HINTS, JOIN, SQL, broadcast
>
> Following query throws HintLoggerError Warnings in v3.1.2.
>
> {code:java}
> // code placeholder
> val Query = " SELECT /*+ BROADCASTJOIN(L1, L2, L3) */ " +
> " L1.v1 AS L1V1 " +
> " L4.* " +
> " FROM L1 " +
> " INNER JOIN L2 ON L2.id = L1.id " +
> " INNER JOIN L3 ON L3.id = L1.id " +
> " LEFT JOIN L4 ON L4.id = L1.id AND L4.idx = L2.idx AND L4.time BETWEEN L3.time1 AND L3.time2 "
> {code}
>
> Following is the warning it thorws during runtime:
> {code:java}
> WARN HintErrorLogger: Count not find relation 'L1' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
> WARN HintErrorLogger: Count not find relation 'L2' specified in hint 'BROADCASTJOIN(L1,L2,L3)'
> WARN HintErrorLogger: Count not find relation 'L3' specified in hint 'BROADCASTJOIN(L1,L2,L3)'{code}
>
> The same query didn't have any warnings in v2.4.7. I am not entirely sure if this is inherently not broadcasting the three small tables ({{{}L1, L2, L3{}}}) when doing a Left Join with a bigger table (L4).
> I have set \{{autoBroadcastJoinThreshold = 4G }} which is way bigger than L1+L2+L3.
>
> Let me know if you need more info.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org