You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@sqoop.apache.org by Ken Krugler <kk...@transpac.com> on 2011/09/20 22:35:40 UTC

Controlling both transaction size and load during

During an import from a large table, we want to avoid using too many mappers, as that would put too much load on the database.

However that winds up generating very large transactions, e.g. 30M+ rows per request.

Which in turn can cause a transaction timeout, if it takes longer than about 3000 seconds.

Is there any way to control both the load (number of parallel requests) and the size of each request?

Thanks,

-- Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr