You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ofbiz.apache.org by Bilgin Ibryam <bi...@gmail.com> on 2010/08/11 11:40:39 UTC
Re: What is a ETL tool in OFBiz?
For reference:
After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found
out that the most easier way to integrate data in Ofbiz is using Talend.
The process:
- Connect Talend to the external DB
- For each row, prepare the data (doing transformations, joins with other
tables, call web services to get ISO codes)
- Call an ofbiz service to create the data in Ofbiz DB (like create product)
Ofbiz services can be called as web service (real pain with the way
parameters are encoded in maps) or through XML-RPC
The process is really easy, since Talend can connect to many data sources,
and importing the data using Ofbiz services ensure that the data is properly
created.
Bilgin
On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <bj...@free-man.net> wrote:
> your welcome
> in addition if you got the DB route you can write scripts to map ofbiz to
> the other db and use SECAS.
> so if say a createcustomers service is run you can have your SECA call your
> script and load the other DB.
> from you legacy db you can have scheduled services that pull from it to
> ofbiz.
>
> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>
> Hi, BJ Freeman
>> Thanks for your quick answer.
>>
>> Regards,
>> Chatree Srichart
>>
>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net> wrote:
>>
>> it depends on the data form you using.
>>> for files
>>>
>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>
>>> if you have a db you can connect to ofbiz then you can
>>> create the entities and write minilang scripts to pull from the old one,
>>> map then store in ofbiz.
>>>
>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>> your need to define a datasource in the entityengine.xml for your db, to
>>> use this tool.
>>>
>>>
>>>
>>>
>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>
>>> Hi community,
>>>
>>>>
>>>> Now I'm working on OFBiz and need to integrate it to several systems. I
>>>> have
>>>> to get data from several sources to store in OFBiz data model.
>>>> I need to use ETL tool for help this process but I don't know what is a
>>>> ETL
>>>> tool that OFBiz use.
>>>>
>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>
>>>> Regards,
>>>> Chatree Srichart
>>>>
>>>>
>>>>
>>
Re: What is a ETL tool in OFBiz?
Posted by BJ Freeman <bj...@free-man.net>.
thanks
Jacques Le Roux sent the following on 10/2/2010 2:47 AM:
> Done at
> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
> I have refactored and updated you work BJ.
> BTW, I recommend Scite and its variation NotePad++ as simple text editor
> http://notepad-plus-plus.org/ NotePad++ is stuffed with really cool
> tools (see TextFx menu options)
>
> Thanks
>
> Jacques
>
> From: "Jacques Le Roux" <ja...@les7arts.com>
>> Thanks Bilgin,
>>
>> I have enough now to write a really usefull FAQ
>>
>> Jacques
>>
>> From: "Bilgin Ibryam" <bi...@gmail.com>
>>> On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <bj...@free-man.net> wrote:
>>>
>>>> actually was not talking about data flat files as a I was about using a
>>>> datasource to connect to a remote DB and inducing the entities for
>>>> it, then
>>>> writing mininlanq to pull in data from the remote db then mapping to
>>>> the
>>>> service and calling the service.
>>>> seems to me a lot simpler.
>>>>
>>>
>>> In my case I didn't have access to the remote DB and also didn't know
>>> the
>>> structure of the data. So it was easier for the client to use Talend for
>>> extracting data (as he alrady knows the data structure of DB) and
>>> send these
>>> data to OFBiz with xml-rpc
>>>
>>> Bilgin
>>>
>>
>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
Done at https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
I have refactored and updated you work BJ.
BTW, I recommend Scite and its variation NotePad++ as simple text editor http://notepad-plus-plus.org/
NotePad++ is stuffed with really cool tools (see TextFx menu options)
Thanks
Jacques
From: "Jacques Le Roux" <ja...@les7arts.com>
> Thanks Bilgin,
>
> I have enough now to write a really usefull FAQ
>
> Jacques
>
> From: "Bilgin Ibryam" <bi...@gmail.com>
>> On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <bj...@free-man.net> wrote:
>>
>>> actually was not talking about data flat files as a I was about using a
>>> datasource to connect to a remote DB and inducing the entities for it, then
>>> writing mininlanq to pull in data from the remote db then mapping to the
>>> service and calling the service.
>>> seems to me a lot simpler.
>>>
>>
>> In my case I didn't have access to the remote DB and also didn't know the
>> structure of the data. So it was easier for the client to use Talend for
>> extracting data (as he alrady knows the data structure of DB) and send these
>> data to OFBiz with xml-rpc
>>
>> Bilgin
>>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
Thanks Bilgin,
I have enough now to write a really usefull FAQ
Jacques
From: "Bilgin Ibryam" <bi...@gmail.com>
> On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <bj...@free-man.net> wrote:
>
>> actually was not talking about data flat files as a I was about using a
>> datasource to connect to a remote DB and inducing the entities for it, then
>> writing mininlanq to pull in data from the remote db then mapping to the
>> service and calling the service.
>> seems to me a lot simpler.
>>
>
> In my case I didn't have access to the remote DB and also didn't know the
> structure of the data. So it was easier for the client to use Talend for
> extracting data (as he alrady knows the data structure of DB) and send these
> data to OFBiz with xml-rpc
>
> Bilgin
>
Re: What is a ETL tool in OFBiz?
Posted by Bilgin Ibryam <bi...@gmail.com>.
On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <bj...@free-man.net> wrote:
> actually was not talking about data flat files as a I was about using a
> datasource to connect to a remote DB and inducing the entities for it, then
> writing mininlanq to pull in data from the remote db then mapping to the
> service and calling the service.
> seems to me a lot simpler.
>
In my case I didn't have access to the remote DB and also didn't know the
structure of the data. So it was easier for the client to use Talend for
extracting data (as he alrady knows the data structure of DB) and send these
data to OFBiz with xml-rpc
Bilgin
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
I did actually the same. I should have separated my sentences in sections.
2 st lines were about flat files ("when you don't have a legacy or external DB at hand").
Then about the direct DB access ("when relationnal DB(s) is available"). It seems to me also to be the simpler method...
I thought about possible issues with legacy field types. But it's anyway easy to create specific fieldtypes in OFBiz if ever needed.
Jacques
From: "BJ Freeman" <bj...@free-man.net>
> actually was not talking about data flat files as a I was about using a datasource to connect to a remote DB and inducing the
> entities for it, then writing mininlanq to pull in data from the remote db then mapping to the service and calling the service.
> seems to me a lot simpler.
> only draw back is if the remote db is behind a fire wall.
>
>
> Jacques Le Roux sent the following on 9/25/2010 3:15 PM:
>> BJ, Bilgin,
>>
>> I understand that using the OFBiz datafiles tools is good when you don't
>> have a legacy or external DB at hand (ie you have only flat files or such)
>> I remember you already spoke about this method when relationnal DB(s) is
>> available, and it makes sense indeed (simpler you only have to specify
>> the Entity). What would be the drawbacks compared to what Bilgin suggest
>> (fields types, is Talend able to handle triggers or embedded functions
>> effects?, etc.)
>>
>> Thanks
>>
>> Jacques
>>
>> From: "BJ Freeman" <bj...@free-man.net>
>>> I would think if you have two database that do not match that you will
>>> have to do the process at one end or the other of matching the data.\
>>> as well as making sure you have all the requirements of the service
>>> your calling.
>>> where if you connect to the db through ofbiz and setup the entities,
>>> you marshal the data from the remote db and call the service the same
>>> way.
>>> but you cut out any other layers.
>>>
>>> Bilgin Ibryam sent the following on 9/25/2010 11:54 AM:
>>>> Sure. The main advance of using Talend was that it is able to extract
>>>> data
>>>> from many datasources and then invoke ofbiz service thru xml-rpc (or
>>>> soap).
>>>>
>>>> Importing with OFBiz datafiles (or with xml files) requires handling
>>>> of all
>>>> the low-level tasks (primary keys, relatations, types, timestamps) to be
>>>> done manually.
>>>> Consider that you want to import the postal address for a person. You
>>>> have
>>>> to create 5-6 entities and manage the relations, types correctly...
>>>> On the
>>>> other hand, with Talend (or similar tool) you have to extract the needed
>>>> data from legacy data source and call createPartyPostalAddress
>>>> service. No
>>>> need to worry if you missed to populate some of the entities.
>>>>
>>>> Bilgin
>>>>
>>>> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux<
>>>> jacques.le.roux@les7arts.com> wrote:
>>>>
>>>>> Hi Bilgin,
>>>>>
>>>>> Before putting this into the FAQ, I was wondering why you choose to use
>>>>> Talend over OOTB OFBiz datafiles?
>>>>> I guess there are advantages, could you please name some?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jacques
>>>>>
>>>>> From: "Jacques Le Roux"<ja...@les7arts.com>
>>>>>
>>>>> Thanks BJ, Bilgin,
>>>>>>
>>>>>> I will put it in FAQ, except if someone takes the time to dot it...
>>>>>>
>>>>>> Jacques
>>>>>>
>>>>>> From: "Bilgin Ibryam"<bi...@gmail.com>
>>>>>>
>>>>>>> For reference:
>>>>>>>
>>>>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit,
>>>>>>> Talend, we
>>>>>>> found
>>>>>>> out that the most easier way to integrate data in Ofbiz is using
>>>>>>> Talend.
>>>>>>>
>>>>>>> The process:
>>>>>>>
>>>>>>> - Connect Talend to the external DB
>>>>>>> - For each row, prepare the data (doing transformations, joins
>>>>>>> with other
>>>>>>> tables, call web services to get ISO codes)
>>>>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>>>>>> product)
>>>>>>>
>>>>>>> Ofbiz services can be called as web service (real pain with the way
>>>>>>> parameters are encoded in maps) or through XML-RPC
>>>>>>>
>>>>>>> The process is really easy, since Talend can connect to many data
>>>>>>> sources,
>>>>>>> and importing the data using Ofbiz services ensure that the data is
>>>>>>> properly
>>>>>>> created.
>>>>>>>
>>>>>>> Bilgin
>>>>>>>
>>>>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<bj...@free-man.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>> your welcome
>>>>>>>> in addition if you got the DB route you can write scripts to map
>>>>>>>> ofbiz
>>>>>>>> to
>>>>>>>> the other db and use SECAS.
>>>>>>>> so if say a createcustomers service is run you can have your SECA
>>>>>>>> call
>>>>>>>> your
>>>>>>>> script and load the other DB.
>>>>>>>> from you legacy db you can have scheduled services that pull from
>>>>>>>> it to
>>>>>>>> ofbiz.
>>>>>>>>
>>>>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>>>>>
>>>>>>>> Hi, BJ Freeman
>>>>>>>>
>>>>>>>>> Thanks for your quick answer.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Chatree Srichart
>>>>>>>>>
>>>>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> it depends on the data form you using.
>>>>>>>>>
>>>>>>>>>> for files
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>>>>>> create the entities and write minilang scripts to pull from the
>>>>>>>>>> old
>>>>>>>>>> one,
>>>>>>>>>> map then store in ofbiz.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>>>>>>
>>>>>>>>>> your need to define a datasource in the entityengine.xml for
>>>>>>>>>> your db,
>>>>>>>>>> to
>>>>>>>>>> use this tool.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>>>>>
>>>>>>>>>> Hi community,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Now I'm working on OFBiz and need to integrate it to several
>>>>>>>>>>> systems.
>>>>>>>>>>> I
>>>>>>>>>>> have
>>>>>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>>>>>> I need to use ETL tool for help this process but I don't know
>>>>>>>>>>> what is
>>>>>>>>>>> a
>>>>>>>>>>> ETL
>>>>>>>>>>> tool that OFBiz use.
>>>>>>>>>>>
>>>>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chatree Srichart
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by BJ Freeman <bj...@free-man.net>.
actually was not talking about data flat files as a I was about using a
datasource to connect to a remote DB and inducing the entities for it,
then writing mininlanq to pull in data from the remote db then mapping
to the service and calling the service.
seems to me a lot simpler.
only draw back is if the remote db is behind a fire wall.
Jacques Le Roux sent the following on 9/25/2010 3:15 PM:
> BJ, Bilgin,
>
> I understand that using the OFBiz datafiles tools is good when you don't
> have a legacy or external DB at hand (ie you have only flat files or such)
> I remember you already spoke about this method when relationnal DB(s) is
> available, and it makes sense indeed (simpler you only have to specify
> the Entity). What would be the drawbacks compared to what Bilgin suggest
> (fields types, is Talend able to handle triggers or embedded functions
> effects?, etc.)
>
> Thanks
>
> Jacques
>
> From: "BJ Freeman" <bj...@free-man.net>
>> I would think if you have two database that do not match that you will
>> have to do the process at one end or the other of matching the data.\
>> as well as making sure you have all the requirements of the service
>> your calling.
>> where if you connect to the db through ofbiz and setup the entities,
>> you marshal the data from the remote db and call the service the same
>> way.
>> but you cut out any other layers.
>>
>> Bilgin Ibryam sent the following on 9/25/2010 11:54 AM:
>>> Sure. The main advance of using Talend was that it is able to extract
>>> data
>>> from many datasources and then invoke ofbiz service thru xml-rpc (or
>>> soap).
>>>
>>> Importing with OFBiz datafiles (or with xml files) requires handling
>>> of all
>>> the low-level tasks (primary keys, relatations, types, timestamps) to be
>>> done manually.
>>> Consider that you want to import the postal address for a person. You
>>> have
>>> to create 5-6 entities and manage the relations, types correctly...
>>> On the
>>> other hand, with Talend (or similar tool) you have to extract the needed
>>> data from legacy data source and call createPartyPostalAddress
>>> service. No
>>> need to worry if you missed to populate some of the entities.
>>>
>>> Bilgin
>>>
>>> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux<
>>> jacques.le.roux@les7arts.com> wrote:
>>>
>>>> Hi Bilgin,
>>>>
>>>> Before putting this into the FAQ, I was wondering why you choose to use
>>>> Talend over OOTB OFBiz datafiles?
>>>> I guess there are advantages, could you please name some?
>>>>
>>>> Thanks
>>>>
>>>> Jacques
>>>>
>>>> From: "Jacques Le Roux"<ja...@les7arts.com>
>>>>
>>>> Thanks BJ, Bilgin,
>>>>>
>>>>> I will put it in FAQ, except if someone takes the time to dot it...
>>>>>
>>>>> Jacques
>>>>>
>>>>> From: "Bilgin Ibryam"<bi...@gmail.com>
>>>>>
>>>>>> For reference:
>>>>>>
>>>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit,
>>>>>> Talend, we
>>>>>> found
>>>>>> out that the most easier way to integrate data in Ofbiz is using
>>>>>> Talend.
>>>>>>
>>>>>> The process:
>>>>>>
>>>>>> - Connect Talend to the external DB
>>>>>> - For each row, prepare the data (doing transformations, joins
>>>>>> with other
>>>>>> tables, call web services to get ISO codes)
>>>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>>>>> product)
>>>>>>
>>>>>> Ofbiz services can be called as web service (real pain with the way
>>>>>> parameters are encoded in maps) or through XML-RPC
>>>>>>
>>>>>> The process is really easy, since Talend can connect to many data
>>>>>> sources,
>>>>>> and importing the data using Ofbiz services ensure that the data is
>>>>>> properly
>>>>>> created.
>>>>>>
>>>>>> Bilgin
>>>>>>
>>>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<bj...@free-man.net>
>>>>>> wrote:
>>>>>>
>>>>>> your welcome
>>>>>>> in addition if you got the DB route you can write scripts to map
>>>>>>> ofbiz
>>>>>>> to
>>>>>>> the other db and use SECAS.
>>>>>>> so if say a createcustomers service is run you can have your SECA
>>>>>>> call
>>>>>>> your
>>>>>>> script and load the other DB.
>>>>>>> from you legacy db you can have scheduled services that pull from
>>>>>>> it to
>>>>>>> ofbiz.
>>>>>>>
>>>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>>>>
>>>>>>> Hi, BJ Freeman
>>>>>>>
>>>>>>>> Thanks for your quick answer.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chatree Srichart
>>>>>>>>
>>>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> it depends on the data form you using.
>>>>>>>>
>>>>>>>>> for files
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>>>>> create the entities and write minilang scripts to pull from the
>>>>>>>>> old
>>>>>>>>> one,
>>>>>>>>> map then store in ofbiz.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>>>>>
>>>>>>>>> your need to define a datasource in the entityengine.xml for
>>>>>>>>> your db,
>>>>>>>>> to
>>>>>>>>> use this tool.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>>>>
>>>>>>>>> Hi community,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Now I'm working on OFBiz and need to integrate it to several
>>>>>>>>>> systems.
>>>>>>>>>> I
>>>>>>>>>> have
>>>>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>>>>> I need to use ETL tool for help this process but I don't know
>>>>>>>>>> what is
>>>>>>>>>> a
>>>>>>>>>> ETL
>>>>>>>>>> tool that OFBiz use.
>>>>>>>>>>
>>>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chatree Srichart
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
BJ, Bilgin,
I understand that using the OFBiz datafiles tools is good when you don't have a legacy or external DB at hand (ie you have only flat
files or such)
I remember you already spoke about this method when relationnal DB(s) is available, and it makes sense indeed (simpler you only have
to specify the Entity). What would be the drawbacks compared to what Bilgin suggest (fields types, is Talend able to handle triggers
or embedded functions effects?, etc.)
Thanks
Jacques
From: "BJ Freeman" <bj...@free-man.net>
>I would think if you have two database that do not match that you will have to do the process at one end or the other of matching
>the data.\
> as well as making sure you have all the requirements of the service your calling.
> where if you connect to the db through ofbiz and setup the entities,
> you marshal the data from the remote db and call the service the same way.
> but you cut out any other layers.
>
> Bilgin Ibryam sent the following on 9/25/2010 11:54 AM:
>> Sure. The main advance of using Talend was that it is able to extract data
>> from many datasources and then invoke ofbiz service thru xml-rpc (or soap).
>>
>> Importing with OFBiz datafiles (or with xml files) requires handling of all
>> the low-level tasks (primary keys, relatations, types, timestamps) to be
>> done manually.
>> Consider that you want to import the postal address for a person. You have
>> to create 5-6 entities and manage the relations, types correctly... On the
>> other hand, with Talend (or similar tool) you have to extract the needed
>> data from legacy data source and call createPartyPostalAddress service. No
>> need to worry if you missed to populate some of the entities.
>>
>> Bilgin
>>
>> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux<
>> jacques.le.roux@les7arts.com> wrote:
>>
>>> Hi Bilgin,
>>>
>>> Before putting this into the FAQ, I was wondering why you choose to use
>>> Talend over OOTB OFBiz datafiles?
>>> I guess there are advantages, could you please name some?
>>>
>>> Thanks
>>>
>>> Jacques
>>>
>>> From: "Jacques Le Roux"<ja...@les7arts.com>
>>>
>>> Thanks BJ, Bilgin,
>>>>
>>>> I will put it in FAQ, except if someone takes the time to dot it...
>>>>
>>>> Jacques
>>>>
>>>> From: "Bilgin Ibryam"<bi...@gmail.com>
>>>>
>>>>> For reference:
>>>>>
>>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we
>>>>> found
>>>>> out that the most easier way to integrate data in Ofbiz is using Talend.
>>>>>
>>>>> The process:
>>>>>
>>>>> - Connect Talend to the external DB
>>>>> - For each row, prepare the data (doing transformations, joins with other
>>>>> tables, call web services to get ISO codes)
>>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>>>> product)
>>>>>
>>>>> Ofbiz services can be called as web service (real pain with the way
>>>>> parameters are encoded in maps) or through XML-RPC
>>>>>
>>>>> The process is really easy, since Talend can connect to many data
>>>>> sources,
>>>>> and importing the data using Ofbiz services ensure that the data is
>>>>> properly
>>>>> created.
>>>>>
>>>>> Bilgin
>>>>>
>>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<bj...@free-man.net>
>>>>> wrote:
>>>>>
>>>>> your welcome
>>>>>> in addition if you got the DB route you can write scripts to map ofbiz
>>>>>> to
>>>>>> the other db and use SECAS.
>>>>>> so if say a createcustomers service is run you can have your SECA call
>>>>>> your
>>>>>> script and load the other DB.
>>>>>> from you legacy db you can have scheduled services that pull from it to
>>>>>> ofbiz.
>>>>>>
>>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>>>
>>>>>> Hi, BJ Freeman
>>>>>>
>>>>>>> Thanks for your quick answer.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chatree Srichart
>>>>>>>
>>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>> it depends on the data form you using.
>>>>>>>
>>>>>>>> for files
>>>>>>>>
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>>>
>>>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>>>> create the entities and write minilang scripts to pull from the old
>>>>>>>> one,
>>>>>>>> map then store in ofbiz.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>>>> your need to define a datasource in the entityengine.xml for your db,
>>>>>>>> to
>>>>>>>> use this tool.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>>>
>>>>>>>> Hi community,
>>>>>>>>
>>>>>>>>
>>>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems.
>>>>>>>>> I
>>>>>>>>> have
>>>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>>>> I need to use ETL tool for help this process but I don't know what is
>>>>>>>>> a
>>>>>>>>> ETL
>>>>>>>>> tool that OFBiz use.
>>>>>>>>>
>>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Chatree Srichart
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by BJ Freeman <bj...@free-man.net>.
I would think if you have two database that do not match that you will
have to do the process at one end or the other of matching the data.\
as well as making sure you have all the requirements of the service your
calling.
where if you connect to the db through ofbiz and setup the entities,
you marshal the data from the remote db and call the service the same way.
but you cut out any other layers.
Bilgin Ibryam sent the following on 9/25/2010 11:54 AM:
> Sure. The main advance of using Talend was that it is able to extract data
> from many datasources and then invoke ofbiz service thru xml-rpc (or soap).
>
> Importing with OFBiz datafiles (or with xml files) requires handling of all
> the low-level tasks (primary keys, relatations, types, timestamps) to be
> done manually.
> Consider that you want to import the postal address for a person. You have
> to create 5-6 entities and manage the relations, types correctly... On the
> other hand, with Talend (or similar tool) you have to extract the needed
> data from legacy data source and call createPartyPostalAddress service. No
> need to worry if you missed to populate some of the entities.
>
> Bilgin
>
> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux<
> jacques.le.roux@les7arts.com> wrote:
>
>> Hi Bilgin,
>>
>> Before putting this into the FAQ, I was wondering why you choose to use
>> Talend over OOTB OFBiz datafiles?
>> I guess there are advantages, could you please name some?
>>
>> Thanks
>>
>> Jacques
>>
>> From: "Jacques Le Roux"<ja...@les7arts.com>
>>
>> Thanks BJ, Bilgin,
>>>
>>> I will put it in FAQ, except if someone takes the time to dot it...
>>>
>>> Jacques
>>>
>>> From: "Bilgin Ibryam"<bi...@gmail.com>
>>>
>>>> For reference:
>>>>
>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we
>>>> found
>>>> out that the most easier way to integrate data in Ofbiz is using Talend.
>>>>
>>>> The process:
>>>>
>>>> - Connect Talend to the external DB
>>>> - For each row, prepare the data (doing transformations, joins with other
>>>> tables, call web services to get ISO codes)
>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>>> product)
>>>>
>>>> Ofbiz services can be called as web service (real pain with the way
>>>> parameters are encoded in maps) or through XML-RPC
>>>>
>>>> The process is really easy, since Talend can connect to many data
>>>> sources,
>>>> and importing the data using Ofbiz services ensure that the data is
>>>> properly
>>>> created.
>>>>
>>>> Bilgin
>>>>
>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<bj...@free-man.net>
>>>> wrote:
>>>>
>>>> your welcome
>>>>> in addition if you got the DB route you can write scripts to map ofbiz
>>>>> to
>>>>> the other db and use SECAS.
>>>>> so if say a createcustomers service is run you can have your SECA call
>>>>> your
>>>>> script and load the other DB.
>>>>> from you legacy db you can have scheduled services that pull from it to
>>>>> ofbiz.
>>>>>
>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>>
>>>>> Hi, BJ Freeman
>>>>>
>>>>>> Thanks for your quick answer.
>>>>>>
>>>>>> Regards,
>>>>>> Chatree Srichart
>>>>>>
>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>>> wrote:
>>>>>>
>>>>>> it depends on the data form you using.
>>>>>>
>>>>>>> for files
>>>>>>>
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>>
>>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>>> create the entities and write minilang scripts to pull from the old
>>>>>>> one,
>>>>>>> map then store in ofbiz.
>>>>>>>
>>>>>>>
>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>>> your need to define a datasource in the entityengine.xml for your db,
>>>>>>> to
>>>>>>> use this tool.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>>
>>>>>>> Hi community,
>>>>>>>
>>>>>>>
>>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems.
>>>>>>>> I
>>>>>>>> have
>>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>>> I need to use ETL tool for help this process but I don't know what is
>>>>>>>> a
>>>>>>>> ETL
>>>>>>>> tool that OFBiz use.
>>>>>>>>
>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chatree Srichart
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
Thanks Bilgin,
This will help to feed the FAQ. Notably the example helps much.
Jacques
From: "Bilgin Ibryam" <bi...@gmail.com>
> Sure. The main advance of using Talend was that it is able to extract data
> from many datasources and then invoke ofbiz service thru xml-rpc (or soap).
>
> Importing with OFBiz datafiles (or with xml files) requires handling of all
> the low-level tasks (primary keys, relatations, types, timestamps) to be
> done manually.
> Consider that you want to import the postal address for a person. You have
> to create 5-6 entities and manage the relations, types correctly... On the
> other hand, with Talend (or similar tool) you have to extract the needed
> data from legacy data source and call createPartyPostalAddress service. No
> need to worry if you missed to populate some of the entities.
>
> Bilgin
>
> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux <
> jacques.le.roux@les7arts.com> wrote:
>
>> Hi Bilgin,
>>
>> Before putting this into the FAQ, I was wondering why you choose to use
>> Talend over OOTB OFBiz datafiles?
>> I guess there are advantages, could you please name some?
>>
>> Thanks
>>
>> Jacques
>>
>> From: "Jacques Le Roux" <ja...@les7arts.com>
>>
>> Thanks BJ, Bilgin,
>>>
>>> I will put it in FAQ, except if someone takes the time to dot it...
>>>
>>> Jacques
>>>
>>> From: "Bilgin Ibryam" <bi...@gmail.com>
>>>
>>>> For reference:
>>>>
>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we
>>>> found
>>>> out that the most easier way to integrate data in Ofbiz is using Talend.
>>>>
>>>> The process:
>>>>
>>>> - Connect Talend to the external DB
>>>> - For each row, prepare the data (doing transformations, joins with other
>>>> tables, call web services to get ISO codes)
>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>>> product)
>>>>
>>>> Ofbiz services can be called as web service (real pain with the way
>>>> parameters are encoded in maps) or through XML-RPC
>>>>
>>>> The process is really easy, since Talend can connect to many data
>>>> sources,
>>>> and importing the data using Ofbiz services ensure that the data is
>>>> properly
>>>> created.
>>>>
>>>> Bilgin
>>>>
>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <bj...@free-man.net>
>>>> wrote:
>>>>
>>>> your welcome
>>>>> in addition if you got the DB route you can write scripts to map ofbiz
>>>>> to
>>>>> the other db and use SECAS.
>>>>> so if say a createcustomers service is run you can have your SECA call
>>>>> your
>>>>> script and load the other DB.
>>>>> from you legacy db you can have scheduled services that pull from it to
>>>>> ofbiz.
>>>>>
>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>>
>>>>> Hi, BJ Freeman
>>>>>
>>>>>> Thanks for your quick answer.
>>>>>>
>>>>>> Regards,
>>>>>> Chatree Srichart
>>>>>>
>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>>> wrote:
>>>>>>
>>>>>> it depends on the data form you using.
>>>>>>
>>>>>>> for files
>>>>>>>
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>>
>>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>>> create the entities and write minilang scripts to pull from the old
>>>>>>> one,
>>>>>>> map then store in ofbiz.
>>>>>>>
>>>>>>>
>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>>> your need to define a datasource in the entityengine.xml for your db,
>>>>>>> to
>>>>>>> use this tool.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>>
>>>>>>> Hi community,
>>>>>>>
>>>>>>>
>>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems.
>>>>>>>> I
>>>>>>>> have
>>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>>> I need to use ETL tool for help this process but I don't know what is
>>>>>>>> a
>>>>>>>> ETL
>>>>>>>> tool that OFBiz use.
>>>>>>>>
>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chatree Srichart
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by Bilgin Ibryam <bi...@gmail.com>.
Sure. The main advance of using Talend was that it is able to extract data
from many datasources and then invoke ofbiz service thru xml-rpc (or soap).
Importing with OFBiz datafiles (or with xml files) requires handling of all
the low-level tasks (primary keys, relatations, types, timestamps) to be
done manually.
Consider that you want to import the postal address for a person. You have
to create 5-6 entities and manage the relations, types correctly... On the
other hand, with Talend (or similar tool) you have to extract the needed
data from legacy data source and call createPartyPostalAddress service. No
need to worry if you missed to populate some of the entities.
Bilgin
On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux <
jacques.le.roux@les7arts.com> wrote:
> Hi Bilgin,
>
> Before putting this into the FAQ, I was wondering why you choose to use
> Talend over OOTB OFBiz datafiles?
> I guess there are advantages, could you please name some?
>
> Thanks
>
> Jacques
>
> From: "Jacques Le Roux" <ja...@les7arts.com>
>
> Thanks BJ, Bilgin,
>>
>> I will put it in FAQ, except if someone takes the time to dot it...
>>
>> Jacques
>>
>> From: "Bilgin Ibryam" <bi...@gmail.com>
>>
>>> For reference:
>>>
>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we
>>> found
>>> out that the most easier way to integrate data in Ofbiz is using Talend.
>>>
>>> The process:
>>>
>>> - Connect Talend to the external DB
>>> - For each row, prepare the data (doing transformations, joins with other
>>> tables, call web services to get ISO codes)
>>> - Call an ofbiz service to create the data in Ofbiz DB (like create
>>> product)
>>>
>>> Ofbiz services can be called as web service (real pain with the way
>>> parameters are encoded in maps) or through XML-RPC
>>>
>>> The process is really easy, since Talend can connect to many data
>>> sources,
>>> and importing the data using Ofbiz services ensure that the data is
>>> properly
>>> created.
>>>
>>> Bilgin
>>>
>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <bj...@free-man.net>
>>> wrote:
>>>
>>> your welcome
>>>> in addition if you got the DB route you can write scripts to map ofbiz
>>>> to
>>>> the other db and use SECAS.
>>>> so if say a createcustomers service is run you can have your SECA call
>>>> your
>>>> script and load the other DB.
>>>> from you legacy db you can have scheduled services that pull from it to
>>>> ofbiz.
>>>>
>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>>
>>>> Hi, BJ Freeman
>>>>
>>>>> Thanks for your quick answer.
>>>>>
>>>>> Regards,
>>>>> Chatree Srichart
>>>>>
>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net>
>>>>> wrote:
>>>>>
>>>>> it depends on the data form you using.
>>>>>
>>>>>> for files
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>>
>>>>>> if you have a db you can connect to ofbiz then you can
>>>>>> create the entities and write minilang scripts to pull from the old
>>>>>> one,
>>>>>> map then store in ofbiz.
>>>>>>
>>>>>>
>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>>> your need to define a datasource in the entityengine.xml for your db,
>>>>>> to
>>>>>> use this tool.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>>
>>>>>> Hi community,
>>>>>>
>>>>>>
>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems.
>>>>>>> I
>>>>>>> have
>>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>>> I need to use ETL tool for help this process but I don't know what is
>>>>>>> a
>>>>>>> ETL
>>>>>>> tool that OFBiz use.
>>>>>>>
>>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chatree Srichart
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
Hi Bilgin,
Before putting this into the FAQ, I was wondering why you choose to use Talend over OOTB OFBiz datafiles?
I guess there are advantages, could you please name some?
Thanks
Jacques
From: "Jacques Le Roux" <ja...@les7arts.com>
> Thanks BJ, Bilgin,
>
> I will put it in FAQ, except if someone takes the time to dot it...
>
> Jacques
>
> From: "Bilgin Ibryam" <bi...@gmail.com>
>> For reference:
>>
>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found
>> out that the most easier way to integrate data in Ofbiz is using Talend.
>>
>> The process:
>>
>> - Connect Talend to the external DB
>> - For each row, prepare the data (doing transformations, joins with other
>> tables, call web services to get ISO codes)
>> - Call an ofbiz service to create the data in Ofbiz DB (like create product)
>>
>> Ofbiz services can be called as web service (real pain with the way
>> parameters are encoded in maps) or through XML-RPC
>>
>> The process is really easy, since Talend can connect to many data sources,
>> and importing the data using Ofbiz services ensure that the data is properly
>> created.
>>
>> Bilgin
>>
>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <bj...@free-man.net> wrote:
>>
>>> your welcome
>>> in addition if you got the DB route you can write scripts to map ofbiz to
>>> the other db and use SECAS.
>>> so if say a createcustomers service is run you can have your SECA call your
>>> script and load the other DB.
>>> from you legacy db you can have scheduled services that pull from it to
>>> ofbiz.
>>>
>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>>
>>> Hi, BJ Freeman
>>>> Thanks for your quick answer.
>>>>
>>>> Regards,
>>>> Chatree Srichart
>>>>
>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net> wrote:
>>>>
>>>> it depends on the data form you using.
>>>>> for files
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>>
>>>>> if you have a db you can connect to ofbiz then you can
>>>>> create the entities and write minilang scripts to pull from the old one,
>>>>> map then store in ofbiz.
>>>>>
>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>>> your need to define a datasource in the entityengine.xml for your db, to
>>>>> use this tool.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>>
>>>>> Hi community,
>>>>>
>>>>>>
>>>>>> Now I'm working on OFBiz and need to integrate it to several systems. I
>>>>>> have
>>>>>> to get data from several sources to store in OFBiz data model.
>>>>>> I need to use ETL tool for help this process but I don't know what is a
>>>>>> ETL
>>>>>> tool that OFBiz use.
>>>>>>
>>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>>
>>>>>> Regards,
>>>>>> Chatree Srichart
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>
Re: What is a ETL tool in OFBiz?
Posted by Jacques Le Roux <ja...@les7arts.com>.
Thanks BJ, Bilgin,
I will put it in FAQ, except if someone takes the time to dot it...
Jacques
From: "Bilgin Ibryam" <bi...@gmail.com>
> For reference:
>
> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found
> out that the most easier way to integrate data in Ofbiz is using Talend.
>
> The process:
>
> - Connect Talend to the external DB
> - For each row, prepare the data (doing transformations, joins with other
> tables, call web services to get ISO codes)
> - Call an ofbiz service to create the data in Ofbiz DB (like create product)
>
> Ofbiz services can be called as web service (real pain with the way
> parameters are encoded in maps) or through XML-RPC
>
> The process is really easy, since Talend can connect to many data sources,
> and importing the data using Ofbiz services ensure that the data is properly
> created.
>
> Bilgin
>
> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <bj...@free-man.net> wrote:
>
>> your welcome
>> in addition if you got the DB route you can write scripts to map ofbiz to
>> the other db and use SECAS.
>> so if say a createcustomers service is run you can have your SECA call your
>> script and load the other DB.
>> from you legacy db you can have scheduled services that pull from it to
>> ofbiz.
>>
>> Chatree Srichart sent the following on 7/12/2010 1:05 PM:
>>
>> Hi, BJ Freeman
>>> Thanks for your quick answer.
>>>
>>> Regards,
>>> Chatree Srichart
>>>
>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<bj...@free-man.net> wrote:
>>>
>>> it depends on the data form you using.
>>>> for files
>>>>
>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
>>>>
>>>> if you have a db you can connect to ofbiz then you can
>>>> create the entities and write minilang scripts to pull from the old one,
>>>> map then store in ofbiz.
>>>>
>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb
>>>> your need to define a datasource in the entityengine.xml for your db, to
>>>> use this tool.
>>>>
>>>>
>>>>
>>>>
>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM:
>>>>
>>>> Hi community,
>>>>
>>>>>
>>>>> Now I'm working on OFBiz and need to integrate it to several systems. I
>>>>> have
>>>>> to get data from several sources to store in OFBiz data model.
>>>>> I need to use ETL tool for help this process but I don't know what is a
>>>>> ETL
>>>>> tool that OFBiz use.
>>>>>
>>>>> Can anyone tell me what is a ETL tool in OFBiz?
>>>>>
>>>>> Regards,
>>>>> Chatree Srichart
>>>>>
>>>>>
>>>>>
>>>
>