You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@wicket.apache.org by Chris Merrill <ch...@webperformance.com> on 2011/09/15 15:16:03 UTC
GAE, DataView and performance
I'm using Objectify for datastore access in GAE, though the problem would
be the same if I was using the datastore API directly.
For optimal performance, Google recommends doing batch requests to the
datastore whenever possible - they say making the requests in parallel
is much faster than serial. They also recommend retrieving keys instead
of the entire entity whenever you may not need each entity right away -
such as for a paginated list. I've seen some posts indicating that the
performance improvement can be a factory of 10x - e.g. serially fetching
20 items will be 20x a single fetch, but fetching them in parallel
might only be 2x a single fetch.
I'm having difficulty seeing how I would follow those recommendations when
using a DataView with PagingNavigator. My initial query for all the entities
in the list should be for the keys, since only a fraction of the entities will
be displayed at a time. However, since populateItem() is called serially for
each list item, I cannot batch the requests that resolve the keys to their
entities. For best performance, I need to get a callback when a group of items
will be rendered so that I can fetch them all at once - and then call
populateItem() with the resulting items one at a time.
I'm guessing I'll need to dig into DataView and extend it, or perhaps
roll my own implementation?
As an example, here is my current, non-optimal, implementation:
List<Key<MyEntity>> keys = db.getMyEntityKeys();
DataView<Key<MyEntity>> list = new DataView<<Key<MyEntity>>("entity_list", new
ListDataProvider<Key<MyEntity>>(keys)
{
protected void populateItem(final Item<Key<MyEntity>> item)
{
Key<MyEntity> key = item.getModel().getObject();
MyEntity entity = db.getMyEntity(key);
//// add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator("paginator", list);
The optimized version would perhaps look something like the example below,
which resolves all the keys to their entities in one method and then the
populateItem() expects the entity, rather than the key:
List<Key<MyEntity>> keys = db.getMyEntityKeys();
DataView<Key<MyEntity>> list = new DataView<<Key<MyEntity>>("entity_list", new
ListDataProvider<Key<MyEntity>>(keys)
{
protected List<MyEntity> getPageOfEntities(List<Key<MyEntity>> keys)
{
return db.getEntitiesForKeys(keys);
}
protected void populateItem(final Item<MyEntity> item)
{
MyEntity entity = item.getModel().getObject();
//// add components for the entity
}
}
list.setItemsPerPage(20);
add(list);
add(new PagingNavigator("paginator", list);
Anyone have any suggestions how I can do this without essentially re-implementing
DataView? I probably should have looked further into the DataView code before
posting...but the Wicket community seems to have already solved every other problem
I've come across, so I though I'd ask here first :>
TIA!
Chris
--
------------------------------------------------------------------------ -
Chris Merrill | Web Performance, Inc.
chris@webperformance.com | http://webperformance.com
919-433-1762 | 919-845-7601
Web Performance: Website Load Testing Software & Services
------------------------------------------------------------------------ -
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@wicket.apache.org
For additional commands, e-mail: users-help@wicket.apache.org
Re: GAE, DataView and performance
Posted by Chris Merrill <ch...@webperformance.com>.
On 9/15/2011 9:38 AM, manuelbarzi wrote:
> may you execute the query db in IDataProvider.iterator, instead of
> populateItem. see IDataProvider javadoc.
Wow...that was easy! I'm so glad I asked. You guys have thought of
everything.
I love Wicket!
Chris
--
------------------------------------------------------------------------ -
Chris Merrill | Web Performance, Inc.
chris@webperformance.com | http://webperformance.com
919-433-1762 | 919-845-7601
Web Performance: Website Load Testing Software & Services
------------------------------------------------------------------------ -
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@wicket.apache.org
For additional commands, e-mail: users-help@wicket.apache.org
Re: GAE, DataView and performance
Posted by manuelbarzi <ma...@gmail.com>.
may you execute the query db in IDataProvider.iterator, instead of
populateItem. see IDataProvider javadoc.
.
On Thu, Sep 15, 2011 at 3:16 PM, Chris Merrill <ch...@webperformance.com> wrote:
> I'm using Objectify for datastore access in GAE, though the problem would
> be the same if I was using the datastore API directly.
>
> For optimal performance, Google recommends doing batch requests to the
> datastore whenever possible - they say making the requests in parallel
> is much faster than serial. They also recommend retrieving keys instead
> of the entire entity whenever you may not need each entity right away -
> such as for a paginated list. I've seen some posts indicating that the
> performance improvement can be a factory of 10x - e.g. serially fetching
> 20 items will be 20x a single fetch, but fetching them in parallel
> might only be 2x a single fetch.
>
> I'm having difficulty seeing how I would follow those recommendations when
> using a DataView with PagingNavigator. My initial query for all the entities
> in the list should be for the keys, since only a fraction of the entities will
> be displayed at a time. However, since populateItem() is called serially for
> each list item, I cannot batch the requests that resolve the keys to their
> entities. For best performance, I need to get a callback when a group of items
> will be rendered so that I can fetch them all at once - and then call
> populateItem() with the resulting items one at a time.
>
> I'm guessing I'll need to dig into DataView and extend it, or perhaps
> roll my own implementation?
>
>
> As an example, here is my current, non-optimal, implementation:
>
> List<Key<MyEntity>> keys = db.getMyEntityKeys();
> DataView<Key<MyEntity>> list = new DataView<<Key<MyEntity>>("entity_list", new
> ListDataProvider<Key<MyEntity>>(keys)
> {
> protected void populateItem(final Item<Key<MyEntity>> item)
> {
> Key<MyEntity> key = item.getModel().getObject();
> MyEntity entity = db.getMyEntity(key);
> //// add components for the entity
> }
> }
> list.setItemsPerPage(20);
> add(list);
> add(new PagingNavigator("paginator", list);
>
>
> The optimized version would perhaps look something like the example below,
> which resolves all the keys to their entities in one method and then the
> populateItem() expects the entity, rather than the key:
>
> List<Key<MyEntity>> keys = db.getMyEntityKeys();
> DataView<Key<MyEntity>> list = new DataView<<Key<MyEntity>>("entity_list", new
> ListDataProvider<Key<MyEntity>>(keys)
> {
> protected List<MyEntity> getPageOfEntities(List<Key<MyEntity>> keys)
> {
> return db.getEntitiesForKeys(keys);
> }
> protected void populateItem(final Item<MyEntity> item)
> {
> MyEntity entity = item.getModel().getObject();
> //// add components for the entity
> }
> }
> list.setItemsPerPage(20);
> add(list);
> add(new PagingNavigator("paginator", list);
>
>
> Anyone have any suggestions how I can do this without essentially re-implementing
> DataView? I probably should have looked further into the DataView code before
> posting...but the Wicket community seems to have already solved every other problem
> I've come across, so I though I'd ask here first :>
>
> TIA!
> Chris
>
>
> --
> ------------------------------------------------------------------------ -
> Chris Merrill | Web Performance, Inc.
> chris@webperformance.com | http://webperformance.com
> 919-433-1762 | 919-845-7601
>
> Web Performance: Website Load Testing Software & Services
> ------------------------------------------------------------------------ -
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@wicket.apache.org
> For additional commands, e-mail: users-help@wicket.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@wicket.apache.org
For additional commands, e-mail: users-help@wicket.apache.org