about broadcast

[randylu <ra...@gmail.com>]

  In my code, there are two broadcast variables. Sometimes reading the small
one took more time than the big one, so strange!
  Log on slave node is as follows:
Block broadcast_2 stored as values to memory (estimated size *4.0 KB*, free
17.2 GB)
Reading broadcast variable 2 took *9.998537123* s
Block broadcast_1 stored as values to memory (estimated size *705.9 MB*,
free 16.5 GB)
Reading broadcast variable 1 took *2.596005629* s
  Reading the small one took about 0.004s normally, but more then 9s
Occasionally.
  




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/about-broadcast-tp5416.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: about broadcast

[randylu <ra...@gmail.com>]

i found that the small broadcast variable always took about 10s, not 5s or
else.
If there is some property/conf(which is default 10) that control the
timeout?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/about-broadcast-tp5416p5439.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: about broadcast

[randylu <ra...@gmail.com>]

additional, Reading the big broadcast variable always took about 2s.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/about-broadcast-tp5416p5417.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.