You are viewing a plain text version of this content. The canonical link for it is here.
Posted to torque-dev@db.apache.org by Ivano Luberti <lu...@archicoop.it> on 2010/02/11 11:06:02 UTC
newbie and generator rewriting
Hi, as suggested by Thomas I bring this discussion to the dev list.
I'm a newbie so please forgive some clueless observation.
Thomas Fischer ha scritto:
> Ivano ha scritto:
>> ...
>> Unfortunately I'm not familiar with Maven at all but I suspect it is not
>> mandatory to participate in development.
>>
>
> Hm, you want at least maven2 installed and running otherwise it is quite a
> pain to get the project compiling and building. No deeper knowledge is
> required, of course.
>
>
Ok I will try to understand how to use it.
>> I have just subscribed the dev mailing list and I will wait before
>> getting code from the rep.
>> I had already made in the past some little modification to the generator
>> providing it the ability to produce xml schemas only for a subset of the
>> tables in a database.
>> I had added to the generator the ability to read from a configuration
>> file to know which tables get metadata for.
>> However I did that on my own and now I read that generator has been
>> changed a lot.
>> ...
>>
>
> Ah ok, you are using the sql->xml part of the generator ? This is indeed an
> area where noone else of the torque developers is really interested in;
May I ask why ? I had guessed that in the past as a user, but is
something I don't understand.
Is the scenario in which you have to write code for an existing db so
unusual?
The sql->xml is also useful when you change the db and you want to
regenerate the torque classes. How you do that: do you manipulate
manually the XML ?
> we
> could use help in that area. In fact, while reorganizing the generator this
> is a part that still needs rewriting (in my opinion, throw out the
> ant/texen dependency and create front ends for the different build systems.
>
Different build systems ? What are you thinking about besides ant and
maven ?
> Maybe one could also use ddlutils (http://db.apache.org/ddlutils/), I did
> not check this).
I don't know the tool but I have had a look at the web site. In my
opinion the integration makes sense if the ddlutils heavily simplify the
actual code and the tool is widely used (which guarantees is powerful
and stable).
I see that they last released it in 2007, quite a long time.
> If new features appear during this development, the better
> (personally I think the exclusion feature is needed often, to exclude
> system tables and views).
>
Right, in my case I have written a software that uses three different
databases, but two of them are used to import data into the main one.
Import features use only a small subset of the tables of the two dbs, so
no point in generate hundreds of classes when only few of them are used.
> If you are interested in any of that, we should discuss details on the dev
> list.
>
>
As you see I have started doing it , thanks for your introduction
--
==================================================
dott. Ivano Mario Luberti
Archimede Informatica societa' cooperativa a r. l.
Sede Operativa
Via Gereschi 36 - 56126- Pisa
tel.: +39-050- 580959
tel/fax: +39-050-9711344
web: www.archicoop.it
==================================================
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: newbie and generator rewriting
Posted by Ivano Luberti <lu...@archicoop.it>.
I'm guessing if this difference in view come from different experience
or is a matter of skills.
Other comments follow:
Thomas Fischer ha scritto:
>
> Forgive me, but personally I think reverse engineering from the database is
> an inherent pain. You tend to loose important information, e.g.foreign key
> relation (at least using Mysql's MyIsam tables, not so sure for other DBs).
> Also, reverse type mapping can be difficult. In simple cases, you can guess
> the Java type automatically, but you will never get the Torque types
> booleanchar and booleanint back from the DB. And types might be different
> for every supported database. Getting ranges is difficult(can one determine
> that a column is varchar(134) or number(10,2) ? No idea.)
> Then, there's default value, no idea if you get that back. And so on. My
> energy is too limited for this wide field.
>
>
I have used Torque with postgres and foreign key are supported by
reverse engineering.
More generally I had no issue with data type.
I had some with MySQL, but I consider that server running quickly to
decline (not saying Torque should ceas support, just expressing an
opinion here).
On the contrary using some specific type like booleanchar or booleanint
would let me wonder what is going to happen in the db. And I should
explain it to my DB guys that work on writing triggers and db functions
> The feeling between the Torque developers was that the reverse engineered
> file serves as starting point but must be tweaked manually afterwards. But
> I'm happy to be proven wrong....
>
>
In this case another field were the project should improve is
documentation about the XML format and how to use that.
>
>> The sql->xml is also useful when you change the db and you want to
>> regenerate the torque classes. How you do that: do you manipulate
>> manually the XML ?
>>
>
> Yes, and create the alter table script manually. Altering an existing
> database requires good thinking and testing anyway most of the cases, so
> handcoding the SQL is the smallest part.
>
>
Not sure about "the smallest part" another advantage to make changes
directly in the db is that you prevent data loosing in most cases
(adding column, changinng names.......)
>
>
> Only those two :-) But the old generator is strongly tied to ant which is
> quite pointless for the maven part. It is not a big problem, but the code
> is more complicated and heavy than it has to be.
> In my eyes, the better architecture is to provide the functionality in
> plain java (build-system-agnostic) and provide adapters for the different
> build systems.
>
>
No doubt.
--
==================================================
dott. Ivano Mario Luberti
Archimede Informatica societa' cooperativa a r. l.
Sede Operativa
Via Gereschi 36 - 56126- Pisa
tel.: +39-050- 580959
tel/fax: +39-050-9711344
web: www.archicoop.it
==================================================
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: newbie and generator rewriting
Posted by Ivano Luberti <lu...@archicoop.it>.
Ok, Gregg I will read your links...of course I will need some time
Some comments in line
Greg Monroe ha scritto:
>
>
> I agree TF here. It's hard and will always be wrong for someone or some
> version of SQL. Remember Torque has to work across all major DB server.
> But, as TF said, it's a good tool to get a basic snapshot that can be modified
> to match. IMHO, It's part of the nice rapid application development point of
> view that Torque has.
>
> FWIW, if you dig into how you would do this, you find that you're dependent
> on what the JDBC driver returns about the database. If I remember the JDBC
> specs, the meta data you have to depend on has a lot of "holes" in it as far
> as finding the underlying DB structure. Plus, there is a lot of variation
> in how different drivers implement it.
>
>
As I wrote answering to TF, using Postgres I didn't find any hole but
maybe I was not using features to expose them.
>
> I think it's important to consider a couple of cases for SQL updates. The
> first is the major structural changes that requires a lot of testing, etc.
> as TF talks about above.
>
> The second is case is the additive / minor updates one. E.g. I've added more
> tables to support more features and maybe a few columns to existing tables.
> This is a much more straight forward change and sql-xml is very useful in this.
>
>
Exactly
> FWIW, in the latter case, I tend to do is create the SQL build from the
> *schema.xml that has been checked in. Create a new DB with tables from it.
> Do an sql->xml process on the development/test DB and on the new DB. Compare
> the two and resolve any "whoops... I added that column to the DB but not the
> XML. Repeat until they match... (plus keep and eye out for missing FK/Indices).
>
> Then I'll use this XML with the DDLUtils API to create the basic ALTER script.
> It has the ability to do an compare of the real DB to an XML and create a
> nice SQL script. This isn't perfect, but it's a good start and with some
> review and quick editing, give me the script to run during the code upgrade.
>
>
Ah ok: that is a good suggestion. I didn't know about DDL utils api
until TF pointed them to me
> Hope this isn't too confusing since I've been typing on it piece meal over
> the day.
>
>
I think it is quite clear.
Now I have to make room in my schedule to find time to read your material.
I hope to do next week , but I have two deliveries.
==================================================
dott. Ivano Mario Luberti
Archimede Informatica societa' cooperativa a r. l.
Sede Operativa
Via Gereschi 36 - 56126- Pisa
tel.: +39-050- 580959
tel/fax: +39-050-9711344
web: www.archicoop.it
==================================================
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
RE: newbie and generator rewriting
Posted by Greg Monroe <Gr...@dukece.com>.
More comments in line.. but first, one think to note about 4.0 is that we're
also planning to convert the XML file format from DTD to XSD. See:
http://people.apache.org/~gmonroe/torqueSchema/
So when we talk about various tasks needing work, this is part of it. Oh,
and you might want to read some of the following email threads:
http://www.mail-archive.com/torque-dev@db.apache.org/msg04852.html
http://www.mail-archive.com/torque-dev@db.apache.org/msg04890.html
I think there are some other topics around that same time period as well.
As I said.. more comments in line.
> Thomas Fischer said:
> > > ...
> > > Ah ok, you are using the sql->xml part of the generator ? This is
> indeed an
> > > area where noone else of the torque developers is really interested
> in;
>
> > May I ask why ? I had guessed that in the past as a user, but is
> > something I don't understand.
>
> Forgive me, but personally I think reverse engineering from the database
> is
> an inherent pain. You tend to loose important information, e.g.foreign
> key
> relation (at least using Mysql's MyIsam tables, not so sure for other
> DBs).
> Also, reverse type mapping can be difficult. In simple cases, you can
> guess
> the Java type automatically, but you will never get the Torque types
> booleanchar and booleanint back from the DB. And types might be different
> for every supported database. Getting ranges is difficult(can one
> determine
> that a column is varchar(134) or number(10,2) ? No idea.)
> Then, there's default value, no idea if you get that back. And so on. My
> energy is too limited for this wide field.
>
> The feeling between the Torque developers was that the reverse engineered
> file serves as starting point but must be tweaked manually afterwards.
> But
> I'm happy to be proven wrong....
I agree TF here. It's hard and will always be wrong for someone or some
version of SQL. Remember Torque has to work across all major DB server.
But, as TF said, it's a good tool to get a basic snapshot that can be modified
to match. IMHO, It's part of the nice rapid application development point of
view that Torque has.
FWIW, if you dig into how you would do this, you find that you're dependent
on what the JDBC driver returns about the database. If I remember the JDBC
specs, the meta data you have to depend on has a lot of "holes" in it as far
as finding the underlying DB structure. Plus, there is a lot of variation
in how different drivers implement it.
Theoretically, I think the SQL standard has definition for a set of system
tables (e.g. SYSCATELOG, etc) that has detailed DB Schema info. But they
aren't something people strive to keep to standard. (I think Oracle
discourages using them...)
That said, there is probably room for improvement here if someone wants
to dig into it.
>
> > Is the scenario in which you have to write code for an existing db so
> > unusual?
>
> For me, it does not happen often. Usually the projects I do is developing
> from scratch.
I've found myself adding Torque objects on top of other people's DB a lot.
In this case, I tend to say "managing the schema" is the other code's job and
Torque's job is just to let my code interact with the data. In this situation,
the sql->xml plus a little tweaking can easily produce the "no-sql" objects
I need.
>
> > The sql->xml is also useful when you change the db and you want to
> > regenerate the torque classes. How you do that: do you manipulate
> > manually the XML ?
>
> Yes, and create the alter table script manually. Altering an existing
> database requires good thinking and testing anyway most of the cases, so
> handcoding the SQL is the smallest part.
>
I think it's important to consider a couple of cases for SQL updates. The
first is the major structural changes that requires a lot of testing, etc.
as TF talks about above.
The second is case is the additive / minor updates one. E.g. I've added more
tables to support more features and maybe a few columns to existing tables.
This is a much more straight forward change and sql-xml is very useful in this.
FWIW, in the latter case, I tend to do is create the SQL build from the
*schema.xml that has been checked in. Create a new DB with tables from it.
Do an sql->xml process on the development/test DB and on the new DB. Compare
the two and resolve any "whoops... I added that column to the DB but not the
XML. Repeat until they match... (plus keep and eye out for missing FK/Indices).
Then I'll use this XML with the DDLUtils API to create the basic ALTER script.
It has the ability to do an compare of the real DB to an XML and create a
nice SQL script. This isn't perfect, but it's a good start and with some
review and quick editing, give me the script to run during the code upgrade.
One other benefit of this is that since I run my main app on different
production machines, it catches / handles the cases where one production server
has been "patched" with a new feature/db setup to meet a client need prior
to the feature being fully rolled out. The generated alter script will take
this into account.
I don't think this could ever be fully automated but having the various tools
makes it MUCH more faster to do and less error prone.
> > > we
> > > could use help in that area. In fact, while reorganizing the
> generator
> this
> > > is a part that still needs rewriting (in my opinion, throw out the
> > > ant/texen dependency and create front ends for the different build
> systems.
> > >
> > Different build systems ? What are you thinking about besides ant and
> > maven ?
>
> Only those two :-) But the old generator is strongly tied to ant which is
> quite pointless for the maven part. It is not a big problem, but the code
> is more complicated and heavy than it has to be.
> In my eyes, the better architecture is to provide the functionality in
> plain java (build-system-agnostic) and provide adapters for the different
> build systems.
>
Agree... but as a diehard Ant user who only uses Maven if it is required
(like Torque building and testing...lol)... we should NOT drop Ant
functionality just for Maven... but I think there is a happy median here
as long as both cases are considered.
> > > Maybe one could also use ddlutils (http://db.apache.org/ddlutils/), I
> did
> > > not check this).
> > I don't know the tool but I have had a look at the web site. In my
> > opinion the integration makes sense if the ddlutils heavily simplify
> the
> > actual code and the tool is widely used (which guarantees is powerful
> > and stable).
> > I see that they last released it in 2007, quite a long time.
>
> You are right, there is currently not much traffic there, but I'd guess
> Thomas Dudziak is still around. This was an idea to save double work in
> Torque, but I did not check in detail. For example, I'm not sure that
> ddlutils has maven support... But personally I'd leave that decision to
> the
> person who actually does the work.
I think I was the one who proposed using more of the DDLUtils code in 4.0.
The thought here was that it does not make sense to have two projects
doing basically the same thing. I think we should keep the tasks in
Torque, but the underlying code should use the DDLUtils API to do the heavy
lifting.
As TF says, there are DDLUtils people around, if we show the need and provide
the code patches to make the two work together, it will get added. Worst case
is that a Torque committer could make a request to be allowed to commit to the
DDLUtils project (I think we have the rights already, but asking is the proper
way) or maybe request to "adopt" the project if the "owner" has moved on.
Hope this isn't too confusing since I've been typing on it piece meal over
the day.
Greg
DukeCE Privacy Statement:
Please be advised that this e-mail and any files transmitted with
it are confidential communication or may otherwise be privileged or
confidential and are intended solely for the individual or entity
to whom they are addressed. If you are not the intended recipient
you may not rely on the contents of this email or any attachments,
and we ask that you please not read, copy or retransmit this
communication, but reply to the sender and destroy the email, its
contents, and all copies thereof immediately. Any unauthorized
dissemination, distribution or copying of this communication is
strictly prohibited.
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
RE: newbie and generator rewriting
Posted by Thomas Fischer <fi...@seitenbau.net>.
> > ...
> > Ah ok, you are using the sql->xml part of the generator ? This is
indeed an
> > area where noone else of the torque developers is really interested in;
> May I ask why ? I had guessed that in the past as a user, but is
> something I don't understand.
Forgive me, but personally I think reverse engineering from the database is
an inherent pain. You tend to loose important information, e.g.foreign key
relation (at least using Mysql's MyIsam tables, not so sure for other DBs).
Also, reverse type mapping can be difficult. In simple cases, you can guess
the Java type automatically, but you will never get the Torque types
booleanchar and booleanint back from the DB. And types might be different
for every supported database. Getting ranges is difficult(can one determine
that a column is varchar(134) or number(10,2) ? No idea.)
Then, there's default value, no idea if you get that back. And so on. My
energy is too limited for this wide field.
The feeling between the Torque developers was that the reverse engineered
file serves as starting point but must be tweaked manually afterwards. But
I'm happy to be proven wrong....
> Is the scenario in which you have to write code for an existing db so
> unusual?
For me, it does not happen often. Usually the projects I do is developing
from scratch.
> The sql->xml is also useful when you change the db and you want to
> regenerate the torque classes. How you do that: do you manipulate
> manually the XML ?
Yes, and create the alter table script manually. Altering an existing
database requires good thinking and testing anyway most of the cases, so
handcoding the SQL is the smallest part.
> > we
> > could use help in that area. In fact, while reorganizing the generator
this
> > is a part that still needs rewriting (in my opinion, throw out the
> > ant/texen dependency and create front ends for the different build
systems.
> >
> Different build systems ? What are you thinking about besides ant and
> maven ?
Only those two :-) But the old generator is strongly tied to ant which is
quite pointless for the maven part. It is not a big problem, but the code
is more complicated and heavy than it has to be.
In my eyes, the better architecture is to provide the functionality in
plain java (build-system-agnostic) and provide adapters for the different
build systems.
> > Maybe one could also use ddlutils (http://db.apache.org/ddlutils/), I
did
> > not check this).
> I don't know the tool but I have had a look at the web site. In my
> opinion the integration makes sense if the ddlutils heavily simplify the
> actual code and the tool is widely used (which guarantees is powerful
> and stable).
> I see that they last released it in 2007, quite a long time.
You are right, there is currently not much traffic there, but I'd guess
Thomas Dudziak is still around. This was an idea to save double work in
Torque, but I did not check in detail. For example, I'm not sure that
ddlutils has maven support... But personally I'd leave that decision to the
person who actually does the work.
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org