You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Manas <ma...@hotmail.com> on 2016/03/24 14:00:39 UTC
Create one DB connection per executor
I understand that using foreachPartition I can create one DB connection per
partition level. Is there a way to create a DB connection per executor level
and share that for all partitions/tasks run within that executor? One
approach I am thinking is to have a singleton with say a getConnection
method. The connection object is not created in the driver rather it passes
to the the singleton object the DB connection detail (host, port, user,
password etc). In the foreachPartition this singleton object is passed too.
The getConnection method of the singleton creates the actual connection
object only the first time it's called and returns the same connection
instance for all later invocations. I believe that way each executor JVM
will have one instance of the singleton/connection and thus all
partitions/tasks running within that executor would share the same
connection. I'd like to validate this approach with the spark experts. Does
it have any inherent flaw or is there a better way to create one instance of
an object per executor?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Create-one-DB-connection-per-executor-tp26588.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Create one DB connection per executor
Posted by Gerard Maas <ge...@gmail.com>.
Hi Manas,
The approach is correct, with one caveat: You may have several tasks
executing in parallel in one executor. Having one single connection per JVM
will either fail, if the connection is not thread-safe or become a
bottleneck b/c all task will be competing for the same resource.
The best approach would be to extend your current idea with a pool of
connections, where you can 'borrow' a connection and return it after use.
-kr, Gerard.
On Thu, Mar 24, 2016 at 2:00 PM, Manas <ma...@hotmail.com> wrote:
> I understand that using foreachPartition I can create one DB connection per
> partition level. Is there a way to create a DB connection per executor
> level
> and share that for all partitions/tasks run within that executor? One
> approach I am thinking is to have a singleton with say a getConnection
> method. The connection object is not created in the driver rather it passes
> to the the singleton object the DB connection detail (host, port, user,
> password etc). In the foreachPartition this singleton object is passed too.
> The getConnection method of the singleton creates the actual connection
> object only the first time it's called and returns the same connection
> instance for all later invocations. I believe that way each executor JVM
> will have one instance of the singleton/connection and thus all
> partitions/tasks running within that executor would share the same
> connection. I'd like to validate this approach with the spark experts. Does
> it have any inherent flaw or is there a better way to create one instance
> of
> an object per executor?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Create-one-DB-connection-per-executor-tp26588.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Create one DB connection per executor
Posted by Manas <ma...@hotmail.com>.
Thanks much Gerard & Manas for your inputs. I'll keep in mind the connection
pooling part.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Create-one-DB-connection-per-executor-tp26588p26601.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Create one DB connection per executor
Posted by manasdebashiskar <po...@gmail.com>.
You are on the right track.
The only thing you will have to take care is when two of your partitions try
to access the same connection at the same time.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Create-one-DB-connection-per-executor-tp26588p26593.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org