You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-java@ibatis.apache.org by luy <em...@encs.concordia.ca> on 2008/07/22 17:52:15 UTC
ibatis+springframework for millions data loading
Greeting,
I will load millions of data from Oracle10 to postgreSQL8 through JDBC.
I like the spring+ibatis framework. May I know how ibatis deal with big
chunk of data please?
The example I had is the following, would you suggest better solution?
(1)Transaction begins
(2) select from oracle.table1 into object1
. possible solution is to split table1 into
table1_subgroupdata1, ...n
(3) for loop object1
=> insert into postgreSQL
(4) if success commit
else rollback
(5) Transaction ends
Thanks a lot!
--
View this message in context: http://www.nabble.com/ibatis%2Bspringframework-for-millions-data-loading-tp18592540p18592540.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
Re: ibatis+springframework for millions data loading
Posted by Larry Meadors <la...@gmail.com>.
On Tue, Jul 22, 2008 at 10:15 AM, Dragan Jotanovic
<Dr...@diosphere.com> wrote:
> Bad idea. Ibatis needs to load whole resultset and this would be
> impossible in your case (millions of records, you would need a lot of
> memory and time).
I agree with the "bad idea" part. :)
You'd want to use a row handler to do this if you really feel the need to.
Larry
RE: ibatis+springframework for millions data loading
Posted by Dragan Jotanovic <Dr...@DIOSPHERE.com>.
Bad idea. Ibatis needs to load whole resultset and this would be
impossible in your case (millions of records, you would need a lot of
memory and time).
If you are doing migration from oracle to postgresql, I would simply try
export/import, without custom coding.
Or, use just plain JDBC, it would be easier in this case.
You don't need O/R mapping here.
Or, if you really want to do it through ibatis, you would have to
implement some sort of paging.
Dragan
-----Original Message-----
From: luy [mailto:emilu@encs.concordia.ca]
Sent: Tuesday, July 22, 2008 4:52 PM
To: user-java@ibatis.apache.org
Subject: ibatis+springframework for millions data loading
Greeting,
I will load millions of data from Oracle10 to postgreSQL8 through JDBC.
I like the spring+ibatis framework. May I know how ibatis deal with big
chunk of data please?
The example I had is the following, would you suggest better solution?
(1)Transaction begins
(2) select from oracle.table1 into object1
. possible solution is to split table1 into
table1_subgroupdata1, ...n
(3) for loop object1
=> insert into postgreSQL
(4) if success commit
else rollback
(5) Transaction ends
Thanks a lot!
--
View this message in context:
http://www.nabble.com/ibatis%2Bspringframework-for-millions-data-loading
-tp18592540p18592540.html
Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
Re: ibatis+springframework for millions data loading
Posted by Emi Lu <em...@encs.concordia.ca>.
Go it! And I will use plain JDBC.
Thanks a lot for all your inputs!
Jeff Butler wrote:
> Using JDBC/iBATIS for bulk data loads will usually be slower than using your
> database's bulk load utility. See this page from postgresql dicumentation
> for more information:
>
> http://www.postgresql.org/docs/8.1/interactive/populate.html
>
> If you feel you must use iBATIS, then I'd recommend using a batch and
> issuing periodic commits. See the iBATIS developer's guide (page 56) for an
> example. But, this will likely be much slower than using the bulk loader.
>
> Jeff Butler
>
> On Tue, Jul 22, 2008 at 10:52 AM, luy <em...@encs.concordia.ca> wrote:
>
>> Greeting,
>>
>> I will load millions of data from Oracle10 to postgreSQL8 through JDBC.
>>
>> I like the spring+ibatis framework. May I know how ibatis deal with big
>> chunk of data please?
>>
>> The example I had is the following, would you suggest better solution?
>>
>> (1)Transaction begins
>>
>> (2) select from oracle.table1 into object1
>> . possible solution is to split table1 into
>> table1_subgroupdata1, ...n
>>
>> (3) for loop object1
>> => insert into postgreSQL
>>
>> (4) if success commit
>> else rollback
>>
>> (5) Transaction ends
>>
>> Thanks a lot!
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ibatis%2Bspringframework-for-millions-data-loading-tp18592540p18592540.html
>> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>>
>>
>
Re: ibatis+springframework for millions data loading
Posted by Jeff Butler <je...@gmail.com>.
Using JDBC/iBATIS for bulk data loads will usually be slower than using your
database's bulk load utility. See this page from postgresql dicumentation
for more information:
http://www.postgresql.org/docs/8.1/interactive/populate.html
If you feel you must use iBATIS, then I'd recommend using a batch and
issuing periodic commits. See the iBATIS developer's guide (page 56) for an
example. But, this will likely be much slower than using the bulk loader.
Jeff Butler
On Tue, Jul 22, 2008 at 10:52 AM, luy <em...@encs.concordia.ca> wrote:
>
> Greeting,
>
> I will load millions of data from Oracle10 to postgreSQL8 through JDBC.
>
> I like the spring+ibatis framework. May I know how ibatis deal with big
> chunk of data please?
>
> The example I had is the following, would you suggest better solution?
>
> (1)Transaction begins
>
> (2) select from oracle.table1 into object1
> . possible solution is to split table1 into
> table1_subgroupdata1, ...n
>
> (3) for loop object1
> => insert into postgreSQL
>
> (4) if success commit
> else rollback
>
> (5) Transaction ends
>
> Thanks a lot!
>
> --
> View this message in context:
> http://www.nabble.com/ibatis%2Bspringframework-for-millions-data-loading-tp18592540p18592540.html
> Sent from the iBATIS - User - Java mailing list archive at Nabble.com.
>
>