You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oodt.apache.org by "Brian Foster (JIRA)" <ji...@apache.org> on 2010/12/07 01:12:08 UTC

[jira] Created: (OODT-68) Add Number of Product Hits to ProductPage

Add Number of Product Hits to ProductPage
-----------------------------------------

                 Key: OODT-68
                 URL: https://issues.apache.org/jira/browse/OODT-68
             Project: OODT
          Issue Type: Improvement
          Components: file manager
    Affects Versions: 0.1-incubating
         Environment: none
            Reporter: Brian Foster
            Assignee: Brian Foster
            Priority: Minor
             Fix For: 0.2



- By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits

- This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969411#action_12969411 ] 

Brian Foster commented on OODT-68:
----------------------------------


Here is the proposed changed to ProductPage:

public class ProductPage {
   ...
   ...
   private long numOfHits;
   ...
   ...
   public void setNumOfHits(long numOfHits) {
      this.numOfHits = numOfHits;
   }
   public long getNumOfHits() {
      return this.numOfHits;
   }
   ...
   ...
}

. . . then each Catalog will be modified to call setNumOfHits.


> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968745#action_12968745 ] 

Chris A. Mattmann commented on OODT-68:
---------------------------------------

bq. the patch would be backwards compatible . . . setTotalPages(int) should only be used by Catalog implementations and each Catalog was doing its own number of page calculations (i.e. has to be reimplemented for each Catalog implemented) - setTotalPages(int) is the only method which would be dropped . . . a setter method for number of hits would take its place.

I disagree. The fact that we have to *remove* a method that's existed makes the suggested approach *non*-backwards compatible. If it only added setNumberOfHits (and left setTotalPages as-is) then I'd agree, but not if it takes setTotalPages away.

As for the part about each Catalog doing its own number of page calculations, that _is the intention_. Paging should be computed on the Catalog side which is close to the result set list that it can slice up. If we want to streamline this part, we discussed int he past making an AbstractPagedFilemgrCatalog class that implements paging (using helper methods delegated to the subclasses which extend it). I think even think there was a patch floating around at some point about this. 

bq. also with all this new gui development around the filemgr . . . paging is a heavily used feature . . . having to double query just to get the first page and the number of hits i think should be addressed - it greatly affects page load time over large data sets

Hmmm, I haven't seen this at all? We're talking about 2 queries here, not 10, and users are interactively paging through the results anyways. If they have to query to get the first page and then at worst issue *one more query* to find out the last page (in order to understand the # of results on that last page), I'm not sure I understand the issue (especially when page slices are pretty small, usually between 10-20 products in the deployments I've seen). Furthermore, even if they weren't small, user GUI APIs could take advantage of query(...pageNum), which is an existing method that computes all this stuff *on the fm server catalog end*, which is even more efficient.

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969366#action_12969366 ] 

Paul Ramirez commented on OODT-68:
----------------------------------

Having the total number of products available after a query is useful information just to display back to the end user. I don't believe this can be currently calculated from a product page as that information is abstracted away. Would the interface be changed by adding this information as a member variable to the ProductPage? Does that type of information belong in the ProductPage? 

What is total number of results being used for? I can see the use case for informational purpose but is there anything else? If for informational purposes only then there is a way to do that now. If however the answer is that this number is used in some other calculation lets figure that out and put that case out here for discussion. 

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968600#action_12968600 ] 

Chris A. Mattmann commented on OODT-68:
---------------------------------------

Hi Brian:

I'm not sure I agree with this approach. It seems like the changing of an interface that's been around for a long time w/o a real huge benefit. What is the advantage of doing this over say, making getResultListSize public (which I wouldn't necessarily agree with either, but just saying for posture). 

Also let's be clear: how does this require DataSourceCatalog to do anything special other than what it's always done? The types of calculations you're speaking of (computing # of products) are typically computed in user-facing code (either API-level callers of the pagniation API through fmClient, or web-app consumers of the API looking to display this information on a result set page) anyways, so they don't really impose anything on a specific Catalog implementation (DataSource versus Lucene versus DbXML,etc.)

Cheers,
Chris


> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968910#action_12968910 ] 

Brian Foster commented on OODT-68:
----------------------------------


- let's say i leave setTotalPages(int) as is and just add setNumberOfHits(int) and getNumberOfHits() . . . are we cool then?

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969221#action_12969221 ] 

Chris A. Mattmann commented on OODT-68:
---------------------------------------

bq. let's say i leave setTotalPages(int) as is and just add setNumberOfHits(int) and getNumberOfHits() . . . are we cool then?

Possibly, I'd need to see the approach taken in the patch.

bq. and actually . . . in order to get first page of a product type set and number of products in that product type it currently causes 5 queries instead of just one . . . getFirstPage(ProductType) requires two (one to get num of hits and another to get products) . . . 

Umm, let's be clear where these "queries" are occuring. Both the above 2 are happening on the fm server end, close to the catalog implementation. So, they aren't super expensive.

bq. getLastProductPage(ProductType) calls getFirstPage() and then gets products for last page . . . if a query takes 20-30 seconds (typical for large data sets) 

Umm, by whose count is this _typical_? It sounds like an indexing issue to me with the underlying database implementation. If a query is taking 20-30 seconds to do a simple count then there's likely some _other_ issue here. 

bq. that is then 5x30 which is a LONG time for a user to wait for a GUI to load

Agreed, but look at the bounds. 30 by itself is a *long* time to wait for the GUi to load by itself? My feeling is that we could always create an alternative DataSourceCatalog that implemented paging a different way if we had to deal with these performance issues. I'd rather do this than change the *default paging implementation* that's part of the actual architecture that's been around since the fm inception. And yes, changing/adding/removing a field *is* changing the architecture in some way.


> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969240#action_12969240 ] 

Brian Foster commented on OODT-68:
----------------------------------


- on NPP PEATE we have like 400,000 IASI products, which translates to like 400,000x20 rows in the metadata table (where 20 is the number metadata fields per product) . . . this table continues to grow . . . typically the filemgr (using Oracle DB in this case) is being hit by workflow condition queries at the same time . . . 20 seconds is not bad when all that is taken into consideration . . . but when 20x5 is the case for a GUI page, when i can make it only 20 . . . i'm only adding two methods (won't touch the other methods in ProductPage) . . . would be a new member variable and a setter and getter for that variable (i.e. numOfHits, setNumOfHits(int), getNumOfHits() respectively) . . . then the Catalogs would be modified to use the setter method.

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968957#action_12968957 ] 

Brian Foster commented on OODT-68:
----------------------------------


- and actually . . . in order to get first page of a product type set and number of products in that product type it currently causes 5 queries instead of just one . . . getFirstPage(ProductType) requires two (one to get num of hits and another to get products) . . . getLastProductPage(ProductType) calls getFirstPage() and then gets products for last page . . . if a query takes 20-30 seconds (typical for large data sets) that is then 5x30 which is a LONG time for a user to wait for a GUI to load

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann resolved OODT-68.
-----------------------------------

    Resolution: Fixed
      Assignee: Chris A. Mattmann  (was: Brian Foster)

- fixed in r1053803 using the suggested approach.

> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969416#action_12969416 ] 

Chris A. Mattmann commented on OODT-68:
---------------------------------------

Hi Brian,

Thanks. OK I think that's a good compromise, and I can live with those fields being added to ProductPage. +1!!

Thanks for discussing the issue!

Cheers,
Chris


> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969347#action_12969347 ] 

Chris A. Mattmann commented on OODT-68:
---------------------------------------

bq. on NPP PEATE we have like 400,000 IASI products, which translates to like 400,000x20 rows in the metadata table (where 20 is the number metadata fields per product) . . . this table continues to grow

On OCO, at current count, we've got about 700K GOSATL1bFTS products. I just loaded up the ops UI to test this and the page loaded in ~5-10 secs.
On WRM, we've got 9B data point rows across a number of database tables in MySQL and we're able to query them in < 30 seconds

bq. typically the filemgr (using Oracle DB in this case) is being hit by workflow condition queries at the same time . . . 20 seconds is not bad when all that is taken into consideration

One possible mitigation for this would be to stand up a read-only filemgr to load balance the queries to the underlying Oracle DB. This was something I was considering doing on OCO because I didn't want the single filemgr used for processing to get bogged down when an operator wanted to check status as well. Since the only cost is another port taken and a service up and running (with some allocated # of threads), I think it's a small cost to pay to increase some of the performance aspects you're seeing on PEATE.

bq. i'm only adding two methods (won't touch the other methods in ProductPage) . . . would be a new member variable and a setter and getter for that variable (i.e. numOfHits, setNumOfHits(int), getNumOfHits() respectively) . . . then the Catalogs would be modified to use the setter method.

I'm just wondering why we have to add any methods at all. I'm not sure I see the pressing need for changing this API.



> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (OODT-68) Add Number of Product Hits to ProductPage

Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/OODT-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968622#action_12968622 ] 

Brian Foster commented on OODT-68:
----------------------------------


- the patch would be backwards compatible . . . setTotalPages(int) should only be used by Catalog implementations and each Catalog was doing its own number of page calculations (i.e. has to be reimplemented for each Catalog implemented) -- setTotalPages(int) is the only method which would be dropped . . . a setter method for number of hits would take its place.

- also with all this new gui development around the filemgr . . . paging is a heavily used feature . . . having to double query just to get the first page and the number of hits i think should be addressed -- it greatly affects page load time over large data sets





> Add Number of Product Hits to ProductPage
> -----------------------------------------
>
>                 Key: OODT-68
>                 URL: https://issues.apache.org/jira/browse/OODT-68
>             Project: OODT
>          Issue Type: Improvement
>          Components: file manager
>    Affects Versions: 0.1-incubating
>         Environment: none
>            Reporter: Brian Foster
>            Assignee: Brian Foster
>            Priority: Minor
>             Fix For: 0.2
>
>
> - By adding get number of hits the method setTotalPages(int) is no longer needed; getTotalPages will calculate its return value based on page size and total hits
> - This also allows the number of products to be determined by just getting the first page, instead of getting last page which internally requires DataSourceCatalog to get first page and last page

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.