[jira] [Updated] (OFBIZ-10957) Visit/Visitor specific client IPs tracking exclusion

["Giulio Speri (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OFBIZ-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giulio Speri updated OFBIZ-10957:
---------------------------------
    Labels: ip-exclusion tracking visit  (was: )

> Visit/Visitor specific client IPs tracking exclusion
> ----------------------------------------------------
>
>                 Key: OFBIZ-10957
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-10957
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: framework
>    Affects Versions: Release Branch 13.07
>            Reporter: Giulio Speri
>            Assignee: Giulio Speri
>            Priority: Major
>              Labels: ip-exclusion, tracking, visit
>
> I shortly explain the server architecture on which OFBiz is running: hosted by a third party supplier, there are two (virtual) machines where Apache OFBiz 13.07.03 is running behind Apache2 web server (so we have two web fronts).
> On other two different machines there are the database (MariaDB) and HaProxy has a load balancer.
> HaProxy is configured to perform its Health Checks on the backend servers with a Http GET on the Home Page of one of the two sites.
> Visit and Visitor tracking are enabled, for BI and analytics purposes, so we cannot turn them off.
> These two combined things caused the Visit and Visitor tables to explode in dimensions (we counted about 19M records of Visit and about 67M of Visitors, with the 86% of those caused by the load balancer), since each hit of the HaProxy store a Visit and a Visitor record on the db (plus some other record of other entities, like ShoppingList, due to <firstvisit> and <preprocessor> events).
> A bad side effect of this situation, on the long run, is an overall performance degradation, and an increase in webfront unavailability time windows during the day: it's not necessary to say that our customer was not so happy about this.
>  
> The difficult part of figuring out this problem, was that we did not have direct access to HaProxy and DB machines, to check logs. 
>  
> The solution we thought and implemented, was to exclude from Visit/Visitor tracking specific IP addresses (for our case we were interested in HaProxy IP).
> The Visit and Visitor records (along with firstvisit and preprocessor events) are created mainly in the ControlServlet class, using VisitHandler getVisit/getVisitor/getVisitId methods.
>  
> Our idea consist in reading from a .properties file one or more IP addresses we would like to exclude from tracking and then check them against the client ip address the request is coming from.
> If the client ip address is in the "exclusion list", then do not persist visit/visitor and do not run firstvisit events neither for it.
>  
> The idea is quite simple, but we noticed in few days, a meaningful improvement in overall system performance and stability/availability.
>  
> This kind of exclusion could be also useful in case we do not want to track or register internal IP addresses (ie: mainly used for testing).
>  
> However this solution, should be integrated with a service (cron or scheduled in ofbiz) that keeps the number of records in the tables limited (for example keep only the last month of visit/visitor); I think that these two solutions together, could do the job well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)