You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2019/04/26 20:59:00 UTC

[jira] [Commented] (IMPALA-6088) Rack aware broadcast operator

    [ https://issues.apache.org/jira/browse/IMPALA-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827317#comment-16827317 ] 

Ruslan Dautkhanov commented on IMPALA-6088:
-------------------------------------------

Would be also interesting to look if something like Torrent protocol helps here too.
This Torrent protocol can take into account rack location, if available. 

Spark uses this to avoid -query coordinator- Spark Driver to be a network bottleneck 

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala

{quote}

* The driver divides the serialized object into small chunks and
 * stores those chunks in the BlockManager of the driver.

 * On each executor, the executor first attempts to fetch the object from its BlockManager. If
 * it does not exist, it then uses remote fetches to fetch the small chunks from the driver and/or
 * other executors if available. Once it gets the chunks, it puts the chunks in its own
 * BlockManager, ready for other executors to fetch from.

 * This prevents the driver from being the bottleneck in sending out multiple copies of the
 * broadcast data (one per executor).

{quote}

> Rack aware broadcast operator
> -----------------------------
>
>                 Key: IMPALA-6088
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6088
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Distributed Exec
>            Reporter: Mostafa Mokhtar
>            Priority: Major
>
> When conducting large scale experiments on a 6 rack cluster with aggregator core network topology overall cluster bandwidth utilization was limited. 
> With aggregator core networks nodes and racks are not equidistant, which means a broadcast operation can be inefficient as the broadcasting node needs to send the same data N times to each node on a remote rack. 
> Ideally Rowbatches should be sent once per remote rack then a node on each remote rack would broadcast within its rack. 
> Table below represent rack to rack latency for the 90% of operations, ration between best and worst case is 7.3x 
> | ||	va||	vc||	vd1||	vd3||	ve|
> |va|	4,238|	4,290|	9,692|	8,897|	8,208|
> |vc|	9,290|	4,396|	30,952|	13,529|	14,578|
> |vd1|	9,131|	29,066|	4,346|	17,265|	16,849|
> |vd3|	7,409|	15,517|	17,265|	4,370|	4,687|
> |ve|	4,914|	16,894|	16,430|	4,713|	4,472|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org