You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by "Jacques Le Roux (JIRA)" <ji...@apache.org> on 2007/06/24 22:01:26 UTC
[jira] Closed: (OFBIZ-276) Froogle Data Export
[ https://issues.apache.org/jira/browse/OFBIZ-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Le Roux closed OFBIZ-276.
---------------------------------
Resolution: Duplicate
Assignee: Jacques Le Roux
> Froogle Data Export
> -------------------
>
> Key: OFBIZ-276
> URL: https://issues.apache.org/jira/browse/OFBIZ-276
> Project: OFBiz
> Issue Type: New Feature
> Components: product
> Affects Versions: SVN trunk
> Reporter: Marco Risaliti
> Assignee: Jacques Le Roux
> Priority: Minor
> Attachments: DataFile.patch, DataIO_Froogle.patch, DataIO_Froogle2.patch, froogle.php, froogle_patch.patch, ProductExportServices.java
>
>
> This is a copy of the old-jira issue http://jira.undersunconsulting.com/browse/OFBIZ-886 from Andrew Sykes.
> =====================================================
> Attached is a _very_ rough outline of how I think a froogle export would go.
> This needs a lot of work still, but I'm out of time, so hopefully someone else can take it up.
>
>
> All Comments Work Log Change History Sort Order:
> Comment by Andrew Sykes [16/May/06 01:43 PM] [ Permlink ]
> A rough outline
> Comment by rohit [17/May/06 04:44 AM] [ Permlink ]
> hi,
> i am very new to java and ofbiz. i have been using the oscommerce shopping cart for more than 3 years, but fell for the amazing functionality that ofbiz provides. Though oscommerce performs a very small part of the what ofbiz can it still has some very useful and essential functions for webmaster and web marketeers. i am attaching a froogle data exporter from oscommerce site. The file works independent of the oscommerce installation, i mean one need not have oscommerce installed to get it working. It works excellently and i used to set a cron object such that the file was run once a week.
> i hope this file will give an idea of how to develop a froogle data exporter for ofbiz. if i am not wrong we can also tweek this file to retreive the data from ofbiz database and export to froogle. this might be a temporary solution and may require one to have php installed, but i guess it will save valuable time.
> any comments or suggestions are welcome.
> rohit
> Comment by rohit [17/May/06 04:49 AM] [ Permlink ]
> if one needs more information about the file attached, please visit http://www.oscommerce.com/community/contributions,3876/category,all/search,froogle.
> oscommerce is a basic shoppinh cart, but with a very vibrant community, the feature and modules are very useful to any webmaster. if ofbiz could include some ideas from oscommerce, it could improve even better.
> rohit
> Comment by Andrew Sykes [17/May/06 06:39 AM] [ Permlink ]
> Rohit,
> This would be a VERY messy approach, and I'm not entirely convinced it would even work with OFBiz
> The code I've supplied is more than half of what is required to get a froogle export working.
> I have set it up as a service as that allows it to be scheduled using the JobScheduler.
> If you are looking for somewhere to start with OFBiz code, this is as good a place as any :-)
> Comment by David Yoon [17/May/06 01:19 PM] [ Permlink ]
> Hi guys, I'm also working on this issue, although I'm making something a bit more generic so you could add multiple comparison shopping sites. So far I've configured it to work (update, export, upload) for froogle, and shopAmex. Instead of recursing through all the product_categories, I was thinking of using the inventory_item table as the base, since most of the cs sites want the products that you upload to be available. currently I'm flattening the product, inventory_item, category, price, tables into a single export_product table which contains all the fields necessary for the various cs feed sites. The update for this table is taking quite a long time though (like 30 mins, since there are 22k products in the database). Any suggestions?
> d.
> Comment by Andrew Sykes [17/May/06 04:39 PM] [ Permlink ]
> David,
> I'm not sure it's a good idea to use the inventory_item table as it's not actually necessary to use inventory management to run an OFBiz ecommerce site. Some users just won't care about inventory (e.g. JIT ordering). While others will have a thirdparty application for inventory management.
> Andrew
> Comment by Si Chen [18/May/06 11:07 AM] [ Permlink ]
> Hey everybody -
> I'd like to help. Sorry I got to the party late. We did this once before too, but it got entombed in some custom code that's no longer worth extracting. Here is what I'd suggest:
> 1. Andrew' approach is mostly correctly, but I'd rather use productStoreId than webSiteId. Andrew, you are actually using webSiteId to get productStoreId, but productStoreId is more direct. Also very important: you need to pull all your variant products as well, which I don't see, or people would miss most of your real physical products. I definitely don't think InventoryItem is what you should be using, by the way.
> 2. You'll need to configure some links for the products and probabily also an "atc=___" tag for tracking marketing campaigns (so you know if froogle or whatever actually made you sales!) I'd suggest a .properties file.
> 3. The best way to render the output is actually using the content manager to create a FTL template of froogle's layout and then render against data. Look at the DhlServices.java or the GLExportServices in our Financials module. It's much easier this way than hardcoding formats into Java and more maintainable as well should the formats change (as they inevitably do.)
> Let me know if you need any help...
> Comment by Andrew Sykes [18/May/06 12:15 PM] [ Permlink ]
> The reason for using websiteId is that you need it further down to create the links properly. Some comments in my code would have helped here :-)
> I'm not sure that people would necessarily want to spam froogle with all their variants, (for example, imagine a clothes store). Also, variants do not necesssarily belong to any category, so that might lead to problems.
> I totally agree about rendering, again I apologise for not adding this to my comments.
> Comment by Si Chen [18/May/06 12:37 PM] [ Permlink ]
> Andrew,
> I think you're right about the websiteId then. So you set the URL as well in WebSite, like in the standardContentPrefix field?
> Why don't we make variants/no variants a flag so people can choose?
> No problem about the comments.
> Anybody want to work on this and submit one we can put into OFBiz?
> Si
> Comment by Si Chen [18/May/06 12:41 PM] [ Permlink ]
> Rohit,
> So here's a question about your PHP file, totally off topic, but I'm just curious about osCommerce: How does this patch work? You put it somewhere, and somebody downloads the main osCommerce and then this patch? So it's tested with a particular module...but not necessarily others? Do you just put it in your www/ directory and it runs?
> Now, what about licensing--there's no license terms in the header, so what do we assume it's licensed under? GPL? Apache?
> Please don't get me wrong--I'm not saying it's bad--I'm just curious.
> Thanks,
> Si
> Comment by Ray Barlow [18/May/06 12:53 PM] [ Permlink ]
> I have already done some mods to the version Andrew posted, and thrown a screen in front of it. I had registered a Froogle account the weekend before with a view to working on this so the whole thread has come at a good time.
> The mods I've made already include output created using the DataFile class so I won't rush to change that to FTL rendering. I'll try and polish a little more in the next few days and then post it back here although I doubt that it will be fit for commiting at that stage.
> Comment by rohit [18/May/06 12:58 PM] [ Permlink ]
> Hi Si,
> Thanks for asking. Just to give you an idea Oscommerce is an open source shopping cart based on PHP. The software and its modules like the file attached are under GLP license.
> Oscommerce has a basic module and the community members can create new functionality and share with other. The froogle exporteer file is almost independed on oscommerce files, though it depends on the database structure of oscommerce. One can simply upload the file to a webserver and set a cron object for it to run weekly. that is it.
> i mentioned this file, to explore the possibility of something like this for ofbiz, something thats very easy to install and run and that new users as well as existing users of ofbiz can easy use for there websites.
> i have using oscommerce for 3 years and believe me it does not do even 10% of what ofbiz can do, but the ability of community to develop new contributions such as froogle exporter, etc and the relative ease of integrating these contributions into existing webstores, is where i guess oscommerce takes the lead. i am still learning ofbiz and i hope i am proved wrong. And i did like to add that you must visit oscommerce.com and take a look on how the 100K community members have contributed to it.
> Rohit
> Comment by Andrew Sykes [18/May/06 01:49 PM] [ Permlink ]
> I'm glad to hear that someone has been spurred into action by my contribution, thanks for looking into this Ray. You may want to contact Ian Gilbert (if you haven't already) as he was also planning to do something with it, perhaps you guys could collaborate?
> I think Si's suggestion of an optional include variants (may) be useful - can you give an example scenario please Si? Also one thing to consider, there is a filter category (whose name escapes me) that specifies which products to show on a site, would this mess up variant posting?
> I think it would also be good to add an optional categoryId parameter to the service definition and a recurse[Y/N] flag. This would mean that without a categoryId posting would start from the browse root, with a categoryId posting would start from the specified category and if recurse==N posting would be exclusively for that category.
> In my additional comments on the user list, I suggested using the DataFile stuff, as that was the first thing that sprung to mind, I really don't see any objection to carrying on down that route, although someone else might know different...
> Has anyone considered whether setting the Froogle category in the CategoryAttr is a good approach, I'd be interested to hear any comments on this...
> Also, what about the FTP part, it would be nice if OFBiz handled that directly, anyone have any suggestioins?
> Finally if FTP is integrated, we need somewhere to store the FTP address and credentials - any suggestions?
> Comment by Si Chen [18/May/06 02:22 PM] [ Permlink ]
> Hey -
> 1. Why variants: if someone searches for "white shirt" on froogle and all you had was the virtual "shirt" but not variants, you might not get listed.
> 2. No idea about the filter category.
> 3. Recurse - yeah probably a good idea too.
> 4. I still think the content template + render is better than DataFile, but I could be convinced otherwise.
> 5. I think there are some FtpServices.java in ofbiz now. Not sure what it does though.
> Comment by Andrew Sykes [18/May/06 02:43 PM] [ Permlink ]
> Si,
> Sure that could be an issue, but would you really want to send
> 5 colours x 5 neck sizes x 5 chest sizes x 10 braiding colours
> = 1250 products
> (ok, a bit excessive, but you get the point I'm sure)
> I'm guessing froogle would probably take quite a dim view of this especially if they all had the same product image and description.
> Also, having done a few clothing sites, they never have products this generic, because...
> A/ They'd never get a google rank
> B/ The retailer would always find a more descriptive product name.
> "White Shirt" is more likely to be a category name.
> I've been trying to think of a scenario where variants would be useful, but I just don't think product names would ever be generic enough to need this.
> Comment by Si Chen [18/May/06 02:47 PM] [ Permlink ]
> That's why it's a configurable parameter, so if you don't want it, you don't have to use it. It's OK if you don't put it in--I can add it later if we decide to use it.
> Also: FTP address and credentials - use a .properties file in a config/ directory. That's how the freight shipping and payment processors are handled.
> Comment by Si Chen [18/May/06 03:02 PM] [ Permlink ]
> Rohit,
> Thanks for your comments. I actually do know of oscommerce but wasn't quite sure how it worked. Thanks for explaining it.
> Well, we are definitely a community project as well and do indeed accept contributions from a fairly large number of people. I think the difference you find is not only a greater set of features but also a more formal framework and process for building applications. Hopefully this will work for you. All this stuff may seem intimidating at first, but you can definitely learn it (if I can learn it, anybody can learn it), and once you know it, it will make your work very easy.
> Ok, end of my digressions.
> Si
> Comment by Ray Barlow [21/May/06 06:03 AM] [ Permlink ]
> DataFile.patch adds support to write a DataFile with a header row.
> Comment by Ray Barlow [21/May/06 06:10 AM] [ Permlink ]
> I have added a menu on the catalog application called "DataIO" next to "Configurations" and in there is a new screen for Froogle.
> The screen contains comments about what is still required to be done which to be honest still includes a fair amount to be a really cool implementation, but this submission will at least generate the Froogle file. I've only run on the demo data so far but will deploy it on one of my live sites asap and test generating and submitting to Froogle for real.
> Although not complete if this is acceptable then it would be nice to commit a version sooner rather than wait for the holy grail of FTP automation etc.
> Comment by rohit [21/May/06 10:28 PM] [ Permlink ]
> Hi,
> Can you please give some instructions on how to install it. I will surely test it and give my feedback.
> rohit
> Comment by Ray Barlow [22/May/06 06:59 AM] [ Permlink ]
> The two files I attached are created as SVN patch files. Depending on your platform and tool depends on how you apply them and to be honest applying patches is not something I've done a lot of. For Windows if your using tortoise SVN (which is very good) try right click on the root folder and "Apply Patch" from the sub menu. For the command line I think you'd use the "patch" command, but for both I'd suggest a test folder and google would be a good option.
> On a more general note I forgot to mention that I saw the "DataIO" section being an area for control of other features being talked about like Yahoo, Amazon and maybe even some simple catalogue data import/export routines.
> Comment by rohit [24/May/06 09:23 AM] [ Permlink ]
> hi Ray,
> i am sorry but i could not get it working. i was initially using opentaps-0.9.0 but then tried it on SVN using the tortoise SVN client.
> i will really appreciate if you could be more specific about how you went about installing. For. e.g. in which folders are these files to be copied into and if there is any code that needs to be edited manually.
> i am new to ofbiz, hence a little more detailed instructions will help a lot.
> thank,
> rohit
> Comment by rohit [24/May/06 11:43 PM] [ Permlink ]
> hi Ray,
> i was able to get the files install, but i am getting the following error, when i try to use the froogle exporter:
> 3057047[ServiceEventHandler.java:317:ERROR]
> ---- exception report ----------------------------------------------------------
> Service invocation error
> Exception: org.ofbiz.service.GenericServiceException
> Message: Service target threw an unexpected exception (null)
> ---- stack trace ---------------------------------------------------------------
> org.ofbiz.service.GenericServiceException: Service target threw an unexpected exception (null)
> org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:113)
> org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
> org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
> org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
> org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
> org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> java.lang.Thread.run(Thread.java:534)
> java.lang.NullPointerException
> org.ofbiz.product.product.ProductExportServices.productsToCsv(ProductExportServices.java:73)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:324)
> org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:101)
> org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:63)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:340)
> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:214)
> org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:123)
> org.ofbiz.webapp.event.ServiceEventHandler.invoke(ServiceEventHandler.java:301)
> org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:416)
> org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:252)
> org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:191)
> org.ofbiz.webapp.control.ControlServlet.doPost(ControlServlet.java:87)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:615)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:526)
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
> java.lang.Thread.run(Thread.java:534)
> --------------------------------------------------------------------------------
> 3057047[ RequestHandler.java:283:ERROR] Request froogleExport caused an error with the following message: Error calling event: org.ofbiz.webapp.event.EventHandlerException: Service invocation error (null)
> 3057047[ RequestHandler.java:395:INFO ] [RequestHandler.doRequest]: Response is a view.
> 3057047[ RequestHandler.java:547:INFO ] servletName=control, view=Froogle
> 3057109[ UtilXml.java:242:DEBUG] XML Read 0.062s: file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
> 3057109[ ScreenFactory.java:120:INFO ] Got 2 screens in 0.062s from: file:/C:/ofbiz/applications/product/widget/catalog/DataIOScreens.xml
> 3057281[ UtilXml.java:242:DEBUG] XML Read 0.156s: file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
> 3057281[ ScreenFactory.java:120:INFO ] Got 13 screens in 0.156s from: file:/C:/ofbiz/applications/product/widget/catalog/CatalogCommonScreens.xml
> 3057328[ UtilXml.java:242:DEBUG] XML Read 0.031s: file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
> 3057344[ ScreenFactory.java:120:INFO ] Got 8 screens in 0.047s from: file:/C:/ofbiz/framework/common/widget/CommonScreens.xml
> 3057578[ UtilXml.java:242:DEBUG] XML Read 0.047s: file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
> 3057578[ ScreenFactory.java:120:INFO ] Got 9 screens in 0.047s from: file:/C:/ofbiz/applications/product/widget/catalog/CommonScreens.xml
> 3057750[ ServiceDispatcher.java:452:DEBUG] [[Sync service finished- total:0.0,since last(Begin):0.0]] - 'catalog / getProductCategoryAndLimitedMembers'
> 3057812[ ControlServlet.java:258:DEBUG] [[[froogleExport] Done rendering page, Servlet Finished- total:1.125,since last([froogleExport] S...):1.125]]
> 3061609[ ControlServlet.java:98 :INFO ] The character encoding of the request is: [null]. The character encoding we will use for the request and response is: [UTF-8]
> 3061625[ ControlServlet.java:134:DEBUG] [[[froogleExport] Servlet Starting, doing setup- total:0.0,since last(Begin):0.0]]
> 3061625[ ControlServlet.java:183:DEBUG] [[[froogleExport] Setup done, doing Event(s) and View(s)- total:0.0,since last([froogleExport] S...):0.0]]
> 3061625[ RequestHandler.java:212:INFO ] [Processing Request]: froogleExport
> 3061672[ ServiceDispatcher.java:386:DEBUG] [[Sync service failed...- total:0.0,since last(Begin):0.0]] - 'catalog / froogleExport'
> 3061687[ ServiceDispatcher.java:389:ERROR]
> Please advise what might be wrong.
> rohit
> Comment by Ray Barlow [05/Jun/06 12:17 PM] [ Permlink ]
> Attached is an improved version that has a little more error checking and also uses drop down lists to reduce user entry errors, for data such as webSiteId. The NPE exception Rohit was experiencing would have been due to an invalid webSiteId entry.
> Also added support for tracking codes if defined in the marketing module and a check to skip null images for products.
> It would be nice to see this reviewed and committed if possible. DataFile patch still stands as submitted before as I've not changed anything in it that for this update.
> Comment by rohit [21/Jul/06 01:58 AM] [ Permlink ]
> hi,
> i have attached a new patch file to work with nre requirement of google base format. This patch has been created on the latest SVN 424158.
> Comment by rohit [21/Jul/06 02:00 AM] [ Permlink ]
> This patch has all the changes. you need only this file to implement froogle on ofbiz.
> Comment by Marco Risaliti [13/Sep/06 04:12 PM] [ Permlink ]
> May I have to move it to the new jira server ?
> Thanks
> Marco
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.