How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

Hi,

I apologize, if this is the second time my email appears on the forum.
 I didn't see the first one show up after I posted it and this is my
first time posting on this forum.  Also, the first time I posted this
email, there were '3D' characters all in my code example.  I don't
know how they got there, so I removed them.

I'm converting my home grown custom DAO's over to IBatis sqlMaps and
the IBatis DAO's.  I'm finding IBatis a lot simpler to use than my
own.  However, I need to do a lot of batch processing and I need to do
it from the DAO tier.  With my custom DAO manager, I simply passed the
same connection to all the DAO's and batching was pretty transparent
and simple.  For example, I would instantiate my DAO's with the
following code:

private Connection destConnection;
private SchoolDAO schoolDAO;
private MajorDAO majorDAO;
private CourseDAO courseDAO;

destConnection = 
  connectionFactory.getConnection
    (ServiceConnectionFactory.CUSTOM_CONNECTION);

final ServiceConnection serviceConnection = 
  new ServiceConnection(destConnection, true, "MaxDB");

final DAOFactory daoFactory = new DAOFactory(serviceConnection);

schoolDAO = (SchoolDAO) daoFactory.getDAO(DAOFactory.SCHOLE_DAO);
majorDAO =  (MajorDAO) daoFactory.getDAO(DAOFactory.MAJOR_DAO);
courseDAO = (CourseDAO) daoFactory.getDAO(DAOFactory.COURSE_DAO);

I would do insertions, updates or deletes using the given DAO's, for N
(say 1000) iterations.  Then I would use a method like the following
to execute a batch.

private void doBatchUpdate() throws SQLException {
  schoolDAO.executeBatch();
  majorDAO.executeBatch();
  courseDAO.executeBatch();
  destConnection.commit();
  destConnection.close();
}

This has been very powerful for me.  Because no matter how complex my
model and no matter how many DAO's I happen to be using, I can easily
accomplish my task.  I've even moved this capability up to my service
tier.  By passing in a connection at the service tier I have
simplified my code even further because a given service class may one
or more DAO's.

I would like to have the same functionality using IBatis Data Access
Objects.  Currently, I don't understand how to do this using your DAO
Manager.  You have a clear example of how to do batching directly
using SqlMaps.  But, I need to be able to do batching at the DAO tier.

Please, could you explain to me how I may do Batch processing using
IBatis DAO's.

Thank you.

Re: How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

You are most welcome.  

Thank you for helping me to understand the proper use of iBatis Dao.

:-)

Re: How do I do batch processing with IBatis DAO

[Prashanth Sukumaran <pr...@yahoo.com>]

WOW.  Sometimes it is amazing the kind of requirement people have. 

There is so much to learn and see.

Thanks for taking your time replying.

Prashanth Sukumaran.



--- Hycel Taylor <hy...@gmail.com> wrote:

> I forgot to put the daoManager.commit(), in my example. 
> 
> The business requirement:
> 
> Actually, my code is a lot more complicated than the three dao's that
> I am using in my example.  There are actually nine Dao's being used in
> this particular batch.  I'm iterating over 500,000 records and I'm
> committing every 10,000 records.  It's a migration.  It's one of many,
> that is taking the data out of one database from a set of tables that
> are very poorly normalized and placing the data into a new database
> with very clearly defined, normalized domains.  No check pointing is
> necessary since, when the actual migration takes place it will only
> have to happen once.  The full migration will  be on set of around
> 50,000,000 rows of data. It's not a lot of data, but if the batching
> is done right, the entire migration process should take place in under
> 6 hours.  Which is all the time that the system can be down for.
> 



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

I forgot to put the daoManager.commit(), in my example. 

The business requirement:

Actually, my code is a lot more complicated than the three dao's that
I am using in my example.  There are actually nine Dao's being used in
this particular batch.  I'm iterating over 500,000 records and I'm
committing every 10,000 records.  It's a migration.  It's one of many,
that is taking the data out of one database from a set of tables that
are very poorly normalized and placing the data into a new database
with very clearly defined, normalized domains.  No check pointing is
necessary since, when the actual migration takes place it will only
have to happen once.  The full migration will  be on set of around
50,000,000 rows of data. It's not a lot of data, but if the batching
is done right, the entire migration process should take place in under
6 hours.  Which is all the time that the system can be down for.

Re: How do I do batch processing with IBatis DAO

[Prashanth Sukumaran <pr...@yahoo.com>]

Yah cool.  You could do that.  You would want to commit the transaction.

Seeing the way you code i am sure you have considered this, but one question

So all these must be under one transaction and in batch mode.  So, even if one fails you want all
to fail(not commit).  Is that the business requirement?

Rgds

Prashanth Sukumaran.

--- Hycel Taylor <hy...@gmail.com> wrote:

> I want to make sure I'm understanding how to use iBatis batching correctly.
> 
> I understand now that SqlMapDaoTemplate, contains the method,
> startBatch() and execute batch.  My Dao will extend SqlMapBaseDao
> which extends SqlMapDaoTemplate.
> 
> So, since my Dao's are sharing the same daoManager, if I startBatch()
> on each dao, do some work, and then executeBatch() on each dao, the
> set of Dao's will be considered to be apart of the same batch?
> 
> Example: 
> 
> try {
>     daoManager.startTransaction();
> 
>     SqlMapSchoolDao schoolDao = new SqlMapSchoolDao(daoManager);
>     SqlMapMajorDao majorDao = new SqlMapMajorDao(daoManager);
>     SqlMapCourseDao courseDao = new SqlMapCourseDao(daoManager);
>     
>     schoolDao.startBatch();
>     majorDao.startBatch();
>     courseDao.startBatch();
>     
>     // Do some iterative tasks.
>     
>     schoolDao.executeBatch();
>     majorDao.executeBatch();
>     courseDao.executeBatch();
> 
>   } catch (Exception e) {
>     throw new ServiceException(e.getMessage());
>   } finally {
>     daoManager.endTransaction();
>   }
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

I want to make sure I'm understanding how to use iBatis batching correctly.

I understand now that SqlMapDaoTemplate, contains the method,
startBatch() and execute batch.  My Dao will extend SqlMapBaseDao
which extends SqlMapDaoTemplate.

So, since my Dao's are sharing the same daoManager, if I startBatch()
on each dao, do some work, and then executeBatch() on each dao, the
set of Dao's will be considered to be apart of the same batch?

Example: 

try {
    daoManager.startTransaction();

    SqlMapSchoolDao schoolDao = new SqlMapSchoolDao(daoManager);
    SqlMapMajorDao majorDao = new SqlMapMajorDao(daoManager);
    SqlMapCourseDao courseDao = new SqlMapCourseDao(daoManager);
    
    schoolDao.startBatch();
    majorDao.startBatch();
    courseDao.startBatch();
    
    // Do some iterative tasks.
    
    schoolDao.executeBatch();
    majorDao.executeBatch();
    courseDao.executeBatch();

  } catch (Exception e) {
    throw new ServiceException(e.getMessage());
  } finally {
    daoManager.endTransaction();
  }

Re: How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

Good Advice. 

Thanks.

Re: How do I do batch processing with IBatis DAO

[Prashanth Sukumaran <pr...@yahoo.com>]

Hi,

But it is nothing.

public class BaseSqlMapDao extends SqlMapDaoTemplate {

  protected static final int PAGE_SIZE = 4;

  public BaseSqlMapDao(DaoManager daoManager) {
    super(daoManager);
  }

}

you can directly use SqlMapDaoTemplate instead.  I always follow a good habit of adding a base
class for every layer i have so i can have the duplicate code there.  That is all there is to it.

Rgds

Prashanth Sukumaran.

--- Hycel Taylor <hy...@gmail.com> wrote:

> From what I understand, looking at your source code, it appears that
> the batching is handled by, BaseSqlMapDao.
> 
> As best I can tell, this is not a standard IBatis Class.  It's package
> structure is the following:
> 
> java.lang.Object
>   com.ibatis.dao.client.template.DaoTemplate
>       com.ibatis.dao.client.template.SqlMapDaoTemplate
>           com.ibatis.jpetstore.persistence.sqlmapdao.BaseSqlMapDao
> 
> I think I need to download the pet store demo and take a look at this
> BaseSqlMapDao class.
> 
> Thanks!
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: How do I do batch processing with IBatis DAO

[Hycel Taylor <hy...@gmail.com>]

>From what I understand, looking at your source code, it appears that
the batching is handled by, BaseSqlMapDao.

As best I can tell, this is not a standard IBatis Class.  It's package
structure is the following:

java.lang.Object
  com.ibatis.dao.client.template.DaoTemplate
      com.ibatis.dao.client.template.SqlMapDaoTemplate
          com.ibatis.jpetstore.persistence.sqlmapdao.BaseSqlMapDao

I think I need to download the pet store demo and take a look at this
BaseSqlMapDao class.

Thanks!

Re: How do I do batch processing with IBatis DAO

[Prashanth Sukumaran <pr...@yahoo.com>]

Hi Hycel,

I don't know if this will answer your question but i will try.  

My DAO is defined this way 

        public class ShopSqlMapDao extends BaseSqlMapDao implements ShopDao

Just like in the JPetStore application.

When executing in batch mode i do this.

try {
	startTransaction();

	startBatch();
	for (int i = 0; i < dealerFrLines.length; i++) {
		param.put("dealerFrLines", new Integer(dealerFrLines[i]));
		insert("insertShopDFL", param);
	}
	executeBatch();
	
	commitTransaction();

} finally {
	endTransaction();
}

The reason i use transaction is at line

             insert("insertShopDFL", param);

is where the get connection happens to execute the query.  By adding the start and end
transaction, i will be using only one connection and not use different connections to execute each
insert in the batch mode.

Hope this helps

Prashanth Sukumaran.



--- Hycel Taylor <hy...@gmail.com> wrote:

> Hi,
> 
> I apologize, if this is the second time my email appears on the forum.
>  I didn't see the first one show up after I posted it and this is my
> first time posting on this forum.  Also, the first time I posted this
> email, there were '3D' characters all in my code example.  I don't
> know how they got there, so I removed them.
> 
> I'm converting my home grown custom DAO's over to IBatis sqlMaps and
> the IBatis DAO's.  I'm finding IBatis a lot simpler to use than my
> own.  However, I need to do a lot of batch processing and I need to do
> it from the DAO tier.  With my custom DAO manager, I simply passed the
> same connection to all the DAO's and batching was pretty transparent
> and simple.  For example, I would instantiate my DAO's with the
> following code:
> 
> private Connection destConnection;
> private SchoolDAO schoolDAO;
> private MajorDAO majorDAO;
> private CourseDAO courseDAO;
> 
> destConnection = 
>   connectionFactory.getConnection
>     (ServiceConnectionFactory.CUSTOM_CONNECTION);
> 
> final ServiceConnection serviceConnection = 
>   new ServiceConnection(destConnection, true, "MaxDB");
> 
> final DAOFactory daoFactory = new DAOFactory(serviceConnection);
> 
> schoolDAO = (SchoolDAO) daoFactory.getDAO(DAOFactory.SCHOLE_DAO);
> majorDAO =  (MajorDAO) daoFactory.getDAO(DAOFactory.MAJOR_DAO);
> courseDAO = (CourseDAO) daoFactory.getDAO(DAOFactory.COURSE_DAO);
> 
> I would do insertions, updates or deletes using the given DAO's, for N
> (say 1000) iterations.  Then I would use a method like the following
> to execute a batch.
> 
> private void doBatchUpdate() throws SQLException {
>   schoolDAO.executeBatch();
>   majorDAO.executeBatch();
>   courseDAO.executeBatch();
>   destConnection.commit();
>   destConnection.close();
> }
> 
> This has been very powerful for me.  Because no matter how complex my
> model and no matter how many DAO's I happen to be using, I can easily
> accomplish my task.  I've even moved this capability up to my service
> tier.  By passing in a connection at the service tier I have
> simplified my code even further because a given service class may one
> or more DAO's.
> 
> I would like to have the same functionality using IBatis Data Access
> Objects.  Currently, I don't understand how to do this using your DAO
> Manager.  You have a clear example of how to do batching directly
> using SqlMaps.  But, I need to be able to do batching at the DAO tier.
> 
> Please, could you explain to me how I may do Batch processing using
> IBatis DAO's.
> 
> Thank you.
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com