You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@sqoop.apache.org by Nitin kak <ni...@gmail.com> on 2012/05/22 16:15:49 UTC

Configuring the number of rows per insert statement?

Hi All,

     I am actually working on Sqoop export functionality(Hadoop->RDBMS). I
can see that Sqoop supports multi-row insert statements at 100
rows/statement. Is this number configurable? If not, is there some work
going on in this direction.

I found out that each mapper exporting data to RDBMS does it in a separate
transaction. My question is do the transactions commit in parallel on the
RDBMS side?

Thanks in advance,
Nitin Kak

Re: Configuring the number of rows per insert statement?

Posted by Chalcy <ch...@gmail.com>.

Hi Nitin,

I do not know that.  It worked for me for sql server export as well.

Brock suggested setting sqoop.export.statements.per.transaction , try and
see if this works.  I am sure some Netezza users can help you out.

Cheers,
Chalcy

On Wed, May 23, 2012 at 9:56 AM, Nitin kak <ni...@gmail.com> wrote:

> Hi Chalcy,
>
>     Setting the below variable works for MySQL but does not for Netezza.
> In Netezza the number of records per statement is always 1. I can see it in
> the query logs.
> Do you have any information regarding how it works in Netezza?
>
> Thanks,
> Nitin
>
>
> On Tue, May 22, 2012 at 12:42 PM, Nitin kak <ni...@gmail.com> wrote:
>
>> Thats awesome. Thanks Chalcy.
>>
>> On Tue, May 22, 2012 at 11:48 AM, Chalcy <ch...@gmail.com> wrote:
>>
>>> Hi Nitin,
>>>
>>> Using sqoop.export.records.per.statement , you can set the number of
>>> rows desired.
>>>
>>> --Chalcy
>>>
>>> On Tue, May 22, 2012 at 10:15 AM, Nitin kak <ni...@gmail.com>wrote:
>>>
>>>> Hi All,
>>>>
>>>>      I am actually working on Sqoop export
>>>> functionality(Hadoop->RDBMS). I can see that Sqoop supports multi-row
>>>> insert statements at 100 rows/statement. Is this number configurable? If
>>>> not, is there some work
>>>> going on in this direction.
>>>>
>>>> I found out that each mapper exporting data to RDBMS does it in a
>>>> separate transaction. My question is do the transactions commit in parallel
>>>> on the RDBMS side?
>>>>
>>>> Thanks in advance,
>>>> Nitin Kak
>>>>
>>>
>>>
>>
>

Re: Configuring the number of rows per insert statement?

Posted by Nitin kak <ni...@gmail.com>.

Hi Chalcy,

    Setting the below variable works for MySQL but does not for Netezza. In
Netezza the number of records per statement is always 1. I can see it in
the query logs.
Do you have any information regarding how it works in Netezza?

Thanks,
Nitin

On Tue, May 22, 2012 at 12:42 PM, Nitin kak <ni...@gmail.com> wrote:

> Thats awesome. Thanks Chalcy.
>
> On Tue, May 22, 2012 at 11:48 AM, Chalcy <ch...@gmail.com> wrote:
>
>> Hi Nitin,
>>
>> Using sqoop.export.records.per.statement , you can set the number of
>> rows desired.
>>
>> --Chalcy
>>
>> On Tue, May 22, 2012 at 10:15 AM, Nitin kak <ni...@gmail.com>wrote:
>>
>>> Hi All,
>>>
>>>      I am actually working on Sqoop export functionality(Hadoop->RDBMS).
>>> I can see that Sqoop supports multi-row insert statements at 100
>>> rows/statement. Is this number configurable? If not, is there some work
>>> going on in this direction.
>>>
>>> I found out that each mapper exporting data to RDBMS does it in a
>>> separate transaction. My question is do the transactions commit in parallel
>>> on the RDBMS side?
>>>
>>> Thanks in advance,
>>> Nitin Kak
>>>
>>
>>
>

Re: Configuring the number of rows per insert statement?

Posted by Nitin kak <ni...@gmail.com>.

Thats awesome. Thanks Chalcy.

On Tue, May 22, 2012 at 11:48 AM, Chalcy <ch...@gmail.com> wrote:

> Hi Nitin,
>
> Using sqoop.export.records.per.statement , you can set the number of rows
> desired.
>
> --Chalcy
>
> On Tue, May 22, 2012 at 10:15 AM, Nitin kak <ni...@gmail.com> wrote:
>
>> Hi All,
>>
>>      I am actually working on Sqoop export functionality(Hadoop->RDBMS).
>> I can see that Sqoop supports multi-row insert statements at 100
>> rows/statement. Is this number configurable? If not, is there some work
>> going on in this direction.
>>
>> I found out that each mapper exporting data to RDBMS does it in a
>> separate transaction. My question is do the transactions commit in parallel
>> on the RDBMS side?
>>
>> Thanks in advance,
>> Nitin Kak
>>
>
>

Re: Configuring the number of rows per insert statement?

Posted by Chalcy <ch...@gmail.com>.

Hi Nitin,

Using sqoop.export.records.per.statement , you can set the number of rows
desired.

--Chalcy

On Tue, May 22, 2012 at 10:15 AM, Nitin kak <ni...@gmail.com> wrote:

> Hi All,
>
>      I am actually working on Sqoop export functionality(Hadoop->RDBMS). I
> can see that Sqoop supports multi-row insert statements at 100
> rows/statement. Is this number configurable? If not, is there some work
> going on in this direction.
>
> I found out that each mapper exporting data to RDBMS does it in a separate
> transaction. My question is do the transactions commit in parallel on the
> RDBMS side?
>
> Thanks in advance,
> Nitin Kak
>