You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Yousef Lasi <yo...@gmail.com> on 2015/07/09 17:40:53 UTC

max_hash_table_size

We appear to be running out of memory when attempting a large join with an order by clause
   

	[Client-1] INFO org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener - Query failed:
	org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: org.apache.drill.common.exceptions.DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073380040, Incoming batchBytes 630917. maxBytes 1073741824
	Fragment 1:5
 The docs list the max size for hash tables as 

exec.max_hash_table_size 1073741824 Ending size for hash tables. Range: 0 - 1073741824.

 Does this mean that the largest allowable size has been exceeded and the only option is to disable via:
planner.enable_hashjoin

 If so, is there a reason that the max allowable size is 1073741824 bytes?

 Thanks

Re: max_hash_table_size

Posted by Aman Sinha <as...@maprtech.com>.
It sounds like you are hitting memory limits, not the max_hash_table_size
limit.

The max_hash_table_size is 2^30  = 1073741824 buckets (not bytes).   Note
that this limit is enforced for each minor fragment where the hash table is
built, so it should be plenty (although heavy skew in the data could affect
this).



On Thu, Jul 9, 2015 at 8:40 AM, Yousef Lasi <yo...@gmail.com> wrote:

> We appear to be running out of memory when attempting a large join with an
> order by clause
>
>
>         [Client-1] INFO
> org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener - Query
> failed:
>         org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: org.apache.drill.common.exceptions.DrillRuntimeException: Adding
> this batch causes the total size to exceed max allowed size. Current
> runningBytes 1073380040, Incoming batchBytes 630917. maxBytes 1073741824
>         Fragment 1:5
>  The docs list the max size for hash tables as
>
> exec.max_hash_table_size 1073741824 Ending size for hash tables. Range: 0
> - 1073741824.
>
>  Does this mean that the largest allowable size has been exceeded and the
> only option is to disable via:
> planner.enable_hashjoin
>
>  If so, is there a reason that the max allowable size is 1073741824 bytes?
>
>  Thanks
>