You are viewing a plain text version of this content. The canonical link for it is here.
Posted to torque-dev@db.apache.org by Ate Douma <at...@douma.nu> on 2004/12/09 12:04:02 UTC
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Henning P. Schmiedehausen wrote:
> Chris Custine <ch...@gmail.com> writes:
>
>
>>I think you may want to consider these items in the context of fixing
>>the total nightmare of using torque-gen with Postgres as well. I have
>>posted several times about the hoops I jump through to get JS2 and
>>Postgres to work together, and your suggestions look like a good
>>opportunity to fix those problems with torque-gen and Postgres. To be
>>clear, no single project is at fault (Postgres, Torque, or JS2), but
>>some minor bad choices in each are making Postgres support really
>>crappy right now in JS2.
>
>
> The Torque people right over there at torque-dev@db.apache.org would
> be really happy to know what you consider "total nightmare".
>
> (Personally I'm running a PostgreSQL based Project using sequences as
> ID generators with ~4000 tables in 52 schemas inside a database quite
> successful. All tables are generated an maintained by Torque and
> accessed through Torque and Hibernate).
>
> (Folks, if you have trouble with Torque and/or improvements or wishes,
> LET US KNOW!). Not many Torque people scan the lists of <arbitrary
> project>. Just come over to torque-dev or torque-user and tell us. We
> are not only willing to listen, we are even willing to act and improve
> our code to cater the needs of our users. That's what a project is all
> about.
Henning, thanks for the response and for scanning our list. Seems we aren't just
an arbitrary project to you at least ;-)
In my initial message, http://nagoya.apache.org/eyebrowse/ReadMsg?listId=22&msgNo=19387,
I described a few problems we are encountering with Torque right now.
I won't say it to be a "total nightmare". Really not.
But, right now, I don't think Torque is the best tool for the tasks we currently use it for,
nor for a few important additional tasks I would like to implement.
As far as I can see, Torque really shines where its Pear support is needed
(like in Jetspeed-1). But, for our Jetspeed-2 requirements which mainly concern
mapping an abstract schema definition and inserting initial configuration data,
the current torque generator features are not really sufficient, nor flexible enough.
After reviewing the generator and its templates I found the following lacking or missing:
- separate generation and/or suppression of drop table statements
- order of drop, create table statements causing conflicting result for
different database:
- for hsql a detail table must be generated (thus dropped) before the
parent table otherwise a fk constraint violation is thrown
- for mysql (also with innoDB) the foreign keys are generated inline and thus
changing the order doesn't help: we need to drop the child table (or constraints)
by hand, something Torque doesn't help us with
- no support (as far as I could tell) for generating schema update scripts
- no direct execution of ddl/data sql from xml, only through first written out
scripts. For execution at runtime from a portal/servlet application this isn't blocking
but not exactly what I really would like.
- data xml to sql translation is very limited. We at least need proper
date and timestamp (in configurable format) value transformation support, as well as
support for proper BIT mapping (an issue with PostgreSQL which seems to have
been reported several times on the Torque list). Furthermore, I'd like to be able
to use variables for values like $now or $yesterday, especially for date/timestamp
fields.
- overriding only a few generator templates (like the table.vm and tablefk.vm for
to fix the mysql issue) requires us to maintain *all* templates: torque isn't able
to resolve missing templates from the classpath if templates from the file system are to be
used.
I've started reviewing commons-sql this morning. Although it is still in the commons sandbox,
I think it might be a (far) better match for our requirements.
Especially direct ddl/sql execution is provided as well as schema update at runtime (!).
Its datatype mapping support also looks far more extensive (including number, date and
timestamp format support) than what I've seen in Torque. I expect the PostgreSQL problems
Chris Custine has to be far less then we have right now with Torque.
Finally, it also supports (and uses) templating. Although I only see jelly scripts (eek) in the
codebase, the documentation also speaks of velocity support.
I haven't tested it out yet, so it is far too early to conclude it is what I'm looking for,
but it sure does look promising.
No disregard to Torque or its team, but considering our requirements and the current features
supported by Torque, do you think we should stick with it, or maybe better try commons-sql
(or even something of our own which I don't really want if not needed).
We hope to put out the final release Jetspeed-2 early next year, and our current problems with
the M1 release as well during development need to be addressed soon.
If you do think Torque can meet our needs, how much work and time would it cost?
Thanks again for your response and I hope we can help each other out with this.
I just subscribed to the Torque user and dev lists and also sending this response to the latter.
If you respond, I'd appreciate it if you would send it to the jetspeed-dev list too.
Regards,
Ate
>
> Regards
> Henning (Torque committer)
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by "Henning P. Schmiedehausen" <hp...@intermeta.de>.
Roger Ruttimann <ro...@apache.org> writes:
>Just one other issue I encountered with torque and HSQL. Even if the
>schema defines cascading deletes the attribute doesn't show up in
>scripts generated for HSQL.
probably the templates in Torque were written with an older version of
HSQL when cascades were not implemented.
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/
RedHat Certified Engineer -- Jakarta Turbine Development -- hero for hire
Linux, Java, perl, Solaris -- Consulting, Training, Development
What is more important to you...
[ ] Product Security
or [ ] Quality of Sales and Marketing Support
-- actual question from a Microsoft customer survey
---------------------------------------------------------------------
To unsubscribe, e-mail: jetspeed-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jetspeed-dev-help@jakarta.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Roger Ruttimann <ro...@apache.org>.
Just one other issue I encountered with torque and HSQL. Even if the
schema defines cascading deletes the attribute doesn't show up in
scripts generated for HSQL.
Roger
Ate Douma wrote:
> Henning, thanks for your frankly given response.
>
> Several comments further below.
>
> Henning P. Schmiedehausen wrote:
>
>> Ate Douma <at...@douma.nu> writes:
>>
>>
>>> Henning P. Schmiedehausen wrote:
>>
>>
>>> In my initial message,
>>> http://nagoya.apache.org/eyebrowse/ReadMsg?listId=22&msgNo=19387,
>>> I described a few problems we are encountering with Torque right now.
>>
>>
>>
>> Just read it.
>
>
>>> After reviewing the generator and its templates I found the
>>> following lacking or missing:
>>> - separate generation and/or suppression of drop table statements
>>
>>
>>
>> Yep. the drops should be turn-off-able using a property. They are not.
>>
>>
>>> - order of drop, create table statements causing conflicting result for
>>> different database:
>>> - for hsql a detail table must be generated (thus dropped) before the
>>> parent table otherwise a fk constraint violation is thrown
>>> - for mysql (also with innoDB) the foreign keys are generated
>>> inline and thus
>>> changing the order doesn't help: we need to drop the child table
>>> (or constraints)
>>> by hand, something Torque doesn't help us with
>>
>>
>>
>> This is difficult to achieve IMHO because the sequence of the tables
>> is controlled by the Control.vm template and does not vary for the
>> different databases. It _should_ be possible to suppress foreign key
>> checking temporarily thus allowing arbitrary creation sequences with
>> some databases but I am no real wizard here. This needs more input
>> from other developers / users.
>
> I think the suggestion made by Raphaël Luta on the jetspeed list to
> first drop
> all the foreign key constraints would do the trick. Maybe that won't
> be too
> difficult to implement?
>
>>
>>
>>> - no support (as far as I could tell) for generating schema update
>>> scripts
>>
>>
>>
>> Yes. Torque is "all or nothing".
>
> So, schema updates would always need hand coded scripts if we stick
> with Torque if
> I understand you correctly.
>
>>
>>
>>> - no direct execution of ddl/data sql from xml, only through first
>>> written out
>>> scripts. For execution at runtime from a portal/servlet application
>>> this isn't blocking
>>> but not exactly what I really would like.
>>
>>
>>
>> Correct. The whole generator has been designed as an additional
>> wart^W^Waddition to ant. It definitely needs refactoring and
>> personally I want to look into dropping the whole Velocity
>> templating/Texen shebang. Texen has given us nothing but headaches
>> starting with logging and error reporting and ending up with gobbling
>> up memory like mad... Personally, I believe an XSLT solution would be
>> nicer but I sorely lack time to do any work on this ATM.
>
> That would mean a quite a rewrite of the generator I think. I wonder if
> that would be very profitable in the light of the commons-sql already
> able
> to do this...
>
>>
>> Hibernate e.g. has a better solution by generating the creation SQL
>> directly at run-time. We still shine at the source code generation for
>> the classes/peers though.
>>
>>
>>> - data xml to sql translation is very limited. We at least need proper
>>> date and timestamp (in configurable format) value transformation
>>> support, as well as
>>> support for proper BIT mapping (an issue with PostgreSQL which
>>> seems to have
>>> been reported several times on the Torque list). Furthermore, I'd
>>> like to be able
>>> to use variables for values like $now or $yesterday, especially for
>>> date/timestamp
>>> fields.
>>
>>
>>
>> Support for custom functions would be nice, too (talk encrypting
>> passwords...).
>
> Yes. But if I read you correctly, we shouldn't expect those from
> Torque soon?
>
>>
>>
>>> - overriding only a few generator templates (like the table.vm and
>>> tablefk.vm for
>>> to fix the mysql issue) requires us to maintain *all* templates:
>>> torque isn't able
>>> to resolve missing templates from the classpath if templates from
>>> the file system are to be
>>> used.
>>
>>
>>
>> Texen again. I hope to become a velocity committer at some point (it
>> was basically agreed upon at ApacheCon with Geir and Daniel but no CfV
>> yet. :-( ).
>
> :-(
>
>>
>>
>>> I've started reviewing commons-sql this morning. Although it is
>>> still in the commons sandbox,
>>> I think it might be a (far) better match for our requirements.
>>
>>
>>
>> I've read about commons-sql (it will probably move to the DB project
>
> Do you have any more information about that? Probability, timetable?
>
>> (maybe make a short stint through the incubator) soon) yesterday. I
>> haven't looked at it yet, but everyone that I hear talking about it,
>> really likes it. So yes, I will definitely look at it.
>>
>>
>>> Especially direct ddl/sql execution is provided as well as schema
>>> update at runtime (!).
>>> Its datatype mapping support also looks far more extensive
>>> (including number, date and
>>> timestamp format support) than what I've seen in Torque. I expect
>>> the PostgreSQL problems
>>> Chris Custine has to be far less then we have right now with Torque.
>>
>>
>>
>>> Finally, it also supports (and uses) templating. Although I only see
>>> jelly scripts (eek) in the
>>> codebase, the documentation also speaks of velocity support.
>>
>>
>>
>> Even JvZ has admitted that introducing Jelly was a mistake. :-)
>
> ;-)
>
>>
>>
>>> I haven't tested it out yet, so it is far too early to conclude it
>>> is what I'm looking for,
>>> but it sure does look promising.
>>
>>
>>
>>> No disregard to Torque or its team, but considering our requirements
>>> and the current features
>>> supported by Torque, do you think we should stick with it, or maybe
>>> better try commons-sql
>>> (or even something of our own which I don't really want if not needed).
>>
>>
>>
>> Sure. Torque _is_ a smallish project and lacks in some aspects (you
>> did manage to put your fingers on quite a number of sore spots in the
>> generator. :-) ). Bad metadata for the DBs is another thing that I
>> really hate.
>>
>>
>>> We hope to put out the final release Jetspeed-2 early next year, and
>>> our current problems with
>>> the M1 release as well during development need to be addressed soon.
>>> If you do think Torque can meet our needs, how much work and time
>>> would it cost?
>>
>>
>>
>> To clean up everything that you mentioned above? A lot. I freely
>> admit, that for some things that you like to have (update scripts
>> e.g.) Torque simply has not support. It was not really intended for
>> this. For you immediate needs, I cannot give you an estimate or time
>> frame, sorry. I will open a ticket for your points in the TTS so they
>> won't be forgotten.
>
> Thanks, I've already seen them.
>
> But, to be frank as well: what would your suggestion be?
> Stick with Torque and work through all the issues which will take a
> lot of time as you said
> and might even never be resolved fully, or start investigating a
> possible migration to commons-sql?
>
> I'm more than willing to support the Torque team in resolving these
> issues if there is any change
> they can be resolved (within time).
> Also, the initial response from other Jetspeed team members and users
> on the suggestion of
> David Sean Taylor to try to fix the problems with the Torque team
> first is +1.
>
> But, my time is limited and I really would like to keep my primary
> focus on Jetspeed-2 features,
> not get lost in DB configuration problems.
> So I'm not convinced (yet), certainly not after your frankly given
> response on the current state of Torque,
> we should wait and see.
>
> My impression right now is that we have a much closer match with
> commons-sql already and that our
> near future needs (schema updates, direct ddl/sql execution) will
> prove that even more so.
>
> Thanks again for your response.
>
> Regards, Ate
>
>>
>>
>>> Thanks again for your response and I hope we can help each other out
>>> with this.
>>> I just subscribed to the Torque user and dev lists and also sending
>>> this response to the latter.
>>
>>
>>
>> Cool.
>>
>>
>>> If you respond, I'd appreciate it if you would send it to the
>>> jetspeed-dev list too.
>>
>>
>>
>> Let's see if cross-posting with this newsreader works out... :-)
>>
>> Regards
>> Henning
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jetspeed-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jetspeed-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jetspeed-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jetspeed-dev-help@jakarta.apache.org
creation sequence
Posted by Silvio Godenzi <go...@masterpoint.com>.
>>dropping everything in reverse order than it was created should
>>avoid referential integrity violations.
>>
>>
>
>But what is the "creation sequence"?
>
One valid creation sequence can be found out by doing a topological sort
on the DB schema.
> This might be different for
>different databases (consider inline foreign key generation to
>outside).
>
>
Small support in the in the
org.apache.torque.adapter.DB like
public abstract boolean hasOutsideForeignKeys()
would be helpful.
Regards Silvio
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by "Henning P. Schmiedehausen" <hp...@intermeta.de>.
Silvio Godenzi <go...@masterpoint.com> writes:
>Hi,
>dropping everything in reverse order than it was created should
>avoid referential integrity violations.
But what is the "creation sequence"? This might be different for
different databases (consider inline foreign key generation to
outside).
>BTW: I posted this already in march 2002
Average UseNet memory is three months. :-)
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/
RedHat Certified Engineer -- Jakarta Turbine Development -- hero for hire
Linux, Java, perl, Solaris -- Consulting, Training, Development
What is more important to you...
[ ] Product Security
or [ ] Quality of Sales and Marketing Support
-- actual question from a Microsoft customer survey
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Silvio Godenzi <go...@masterpoint.com>.
Hi,
dropping everything in reverse order than it was created should
avoid referential integrity violations.
BTW: I posted this already in march 2002
Regards,
Silvio
> To change this, we would have to change the loop to
>
> for (all tables in db)
> drop constraints (if external)
>
> for (all tables in db)
> drop tables
>
> (these two probably optional
>
> for (all tables in db)
> create tables
>
> for (all tables in db)
> create foreign and primary keys (if external)
>
> Bigger problem would be tables that have foreign key relations. In
> that case, we would need to reorder the table sequence so that no
> foreign key violations happen.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Thomas Vandahl <th...@tewisoft.de>.
Hi,
Thomas Fischer wrote:
> I have also written quite complicated stylesheets, but by far nothing as
> complicated as the Torque generator, and I have spent lot of time in
> debugging simple bugs in these stylesheets. I do not think much would be
> gained if the generator was completely rewritten in xsl. But this is just
> my personal opinion.
I can recommend XMLSpy for debugging of stylesheets. I use it for XSL:FO
and it does the job quite well.
If a rewrite is really planned, I would like to vote for (and contribute
to) the XSLT-option at least for SQL generation, mainly because it would
make it easier to take care of all the different SQL dialects.
On the other hand, the generator works quite well as it is. I'd prefer
some improvements in the present code over a rewrite, such as a Manager
aware delete method (will submit a patch soon!) and some methods for
simplified handling of weak entities.
Bye, Thomas.
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Thomas Fischer <Fi...@seitenbau.net>.
Hi,
Just a few quick comments to "how to rewrite the generator, if one really
does it"
"Henning P. Schmiedehausen" <hp...@intermeta.de> schrieb am 09.12.2004
16:51:06:
> Ate Douma <at...@douma.nu> writes:
>
> >>
> >> Correct. The whole generator has been designed as an additional
> >> wart^W^Waddition to ant. It definitely needs refactoring and
> >> personally I want to look into dropping the whole Velocity
> >> templating/Texen shebang. Texen has given us nothing but headaches
> >> starting with logging and error reporting and ending up with gobbling
> >> up memory like mad... Personally, I believe an XSLT solution would be
> >> nicer but I sorely lack time to do any work on this ATM.
I have worked now for some time with xslt, and I am not sure whether the
generator would get more simple if it was complete rewritten in xslt. In
the applications I have done so far using xslt, the xml is generated inside
java and is then transfomed by xslt to generate html. In this
configuration, one has the choice to do things either in java or in xslt
(for example, I want to display a special text if the user is admin. Either
I can give the xslt the information "the user is admin" and the stylesheet
decides which text is displayed, or I can decide in java which text to
display and pass it directly to the stylesheet). I have found that it is
almost always better to do things in java and rely on xsltc only for the
very last output formatting (i.e. the second solution in the example). The
main reasons are : 1) java offers much more possibilities than xslt and 2)
it is easier to debug java code than xslt stylesheets.
I have also written quite complicated stylesheets, but by far nothing as
complicated as the Torque generator, and I have spent lot of time in
debugging simple bugs in these stylesheets. I do not think much would be
gained if the generator was completely rewritten in xsl. But this is just
my personal opinion.
> >That would mean a quite a rewrite of the generator I think. I wonder if
> >that would be very profitable in the light of the commons-sql already
able
> >to do this...
>
I do not have an opinion at the moment whether the generator should be
rewritten or not, mainly because I never tried to change anything in the
generator so far. But before one rushes to do it, this should be thought
over very carefully. To rewrite the generator completely takes a lot of
effort, and I am not sure whether the torque developer community has the
manpower to achieve a complete rewriting in reasonable time.
But maybe I just should go ahead and look deeper into the generator as it
is now, to get a better understanding of the problems.
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by "Henning P. Schmiedehausen" <hp...@intermeta.de>.
Ate Douma <at...@douma.nu> writes:
>>>- order of drop, create table statements causing conflicting result for
>>> different database:
>>> - for hsql a detail table must be generated (thus dropped) before the
>>> parent table otherwise a fk constraint violation is thrown
>>> - for mysql (also with innoDB) the foreign keys are generated inline and thus
>>> changing the order doesn't help: we need to drop the child table (or constraints)
>>> by hand, something Torque doesn't help us with
>>
>>
>> This is difficult to achieve IMHO because the sequence of the tables
>> is controlled by the Control.vm template and does not vary for the
>> different databases. It _should_ be possible to suppress foreign key
>> checking temporarily thus allowing arbitrary creation sequences with
>> some databases but I am no real wizard here. This needs more input
>> from other developers / users.
>I think the suggestion made by Rapha�l Luta on the jetspeed list to first drop
>all the foreign key constraints would do the trick. Maybe that won't be too
>difficult to implement?
That could probably be done. However it would still mean some
rewriting of the templates, because currently the processing goes like this:
for (all table templates in the database)
- setup parameters
- parse Table.vm for this table
done
with table.vm loading e.g. drop.vm and then build the table itself. As
you have already noticed, some databases (MySQL noticably) build their
foreign key relations inside the table, others (PostgreSQL) outside
with explicit CONSTRAINT statements.
To change this, we would have to change the loop to
for (all tables in db)
drop constraints (if external)
for (all tables in db)
drop tables
(these two probably optional
for (all tables in db)
create tables
for (all tables in db)
create foreign and primary keys (if external)
Bigger problem would be tables that have foreign key relations. In
that case, we would need to reorder the table sequence so that no
foreign key violations happen.
The funny thing is, that for my current project, I have already
written such a table sequencer (because I had exactly the same problem
(though I "only" nuke the table contents and do not drop the tables).
Ah, I do need to spend some time with Torque...
>> Yes. Torque is "all or nothing".
>So, schema updates would always need hand coded scripts if we stick with Torque if
>I understand you correctly.
Schema updates would (as far as I can understand this) be a "diff"
between two revisions of a schema file. I'm very interested in this
and how to do it (if commons-sql can do it, cool). However, at the
moment, Torque cannot do this.
(commons-sql has been incepted by the same people that, at some point,
dropped development on torque. The fact that commons-sql sat in the
Apache CVS for > 2 years before any activity picked up again and in
the meantime, no one worked on it makes me wonder if all the
advertised features really work or if it is just a "Work in
Progress". I don't know, haven't checked it yet).
>>>- no direct execution of ddl/data sql from xml, only through first written out
>>> scripts. For execution at runtime from a portal/servlet application this isn't blocking
>>> but not exactly what I really would like.
>>
>>
>> Correct. The whole generator has been designed as an additional
>> wart^W^Waddition to ant. It definitely needs refactoring and
>> personally I want to look into dropping the whole Velocity
>> templating/Texen shebang. Texen has given us nothing but headaches
>> starting with logging and error reporting and ending up with gobbling
>> up memory like mad... Personally, I believe an XSLT solution would be
>> nicer but I sorely lack time to do any work on this ATM.
>That would mean a quite a rewrite of the generator I think. I wonder if
>that would be very profitable in the light of the commons-sql already able
>to do this...
To tell you the truth: The generator is not really great. It barely
does its job but any extension or additional functionality is really a
PITA. Rewriting it is very high on my list of stuff to do.
[...]
>> I've read about commons-sql (it will probably move to the DB project
>Do you have any more information about that? Probability, timetable?
Probablity: 100%
Timetable: the next few days. The discussion is _now_ happening on the
PMC lists, that's why I read about it.
[...]
>> Even JvZ has admitted that introducing Jelly was a mistake. :-)
>;-)
As far as I can see, Jelly _is_ one of the cornerstones of
commons-sql. This looks better than Velocity in some places. Must look
deeper into this.
[...]
>But, to be frank as well: what would your suggestion be?
>Stick with Torque and work through all the issues which will take a lot of time as you said
>and might even never be resolved fully, or start investigating a possible migration to commons-sql?
I just did a quick look over the code and found:
- commons-sql has been inactive for almost two years
- some activity has picked up this year, but the site
is outdated and the CVS shows not much activity
- It depends on a number of snapshot jars
- it has no release yet.
- it is also (like torque-gen) missing any object types outside
the database/table/column scope (sequence, view..) which are
necessary to model your database fully. Hibernate is better here.
Personally, I wouldn't hold my breath for this. It definitely needs
work (e.g. there is no MySQL 3/4.0/4.1 distinction, same problem as
torque-gen) just as torque-gen.
>But, my time is limited and I really would like to keep my primary focus on Jetspeed-2 features,
>not get lost in DB configuration problems.
>So I'm not convinced (yet), certainly not after your frankly given response on the current state of Torque,
>we should wait and see.
I don't know about the deeper issues aside from the few mails in this
thread. It all boils down to "how deeply is J2 entwined with
Torque". If your answer is "not much", trying something like OJB might
prove to be the better way. Moving from torque-gen to commons-sql
looks more like a sideway move to new problems to me.
>My impression right now is that we have a much closer match with commons-sql already and that our
>near future needs (schema updates, direct ddl/sql execution) will prove that even more so.
As I said above: JMMV.
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/
RedHat Certified Engineer -- Jakarta Turbine Development -- hero for hire
Linux, Java, perl, Solaris -- Consulting, Training, Development
What is more important to you...
[ ] Product Security
or [ ] Quality of Sales and Marketing Support
-- actual question from a Microsoft customer survey
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Ate Douma <at...@douma.nu>.
Henning, thanks for your frankly given response.
Several comments further below.
Henning P. Schmiedehausen wrote:
> Ate Douma <at...@douma.nu> writes:
>
>
>>Henning P. Schmiedehausen wrote:
>
>>In my initial message, http://nagoya.apache.org/eyebrowse/ReadMsg?listId=22&msgNo=19387,
>>I described a few problems we are encountering with Torque right now.
>
>
> Just read it.
>
>>After reviewing the generator and its templates I found the following lacking or missing:
>>- separate generation and/or suppression of drop table statements
>
>
> Yep. the drops should be turn-off-able using a property. They are not.
>
>
>>- order of drop, create table statements causing conflicting result for
>> different database:
>> - for hsql a detail table must be generated (thus dropped) before the
>> parent table otherwise a fk constraint violation is thrown
>> - for mysql (also with innoDB) the foreign keys are generated inline and thus
>> changing the order doesn't help: we need to drop the child table (or constraints)
>> by hand, something Torque doesn't help us with
>
>
> This is difficult to achieve IMHO because the sequence of the tables
> is controlled by the Control.vm template and does not vary for the
> different databases. It _should_ be possible to suppress foreign key
> checking temporarily thus allowing arbitrary creation sequences with
> some databases but I am no real wizard here. This needs more input
> from other developers / users.
I think the suggestion made by Raphaël Luta on the jetspeed list to first drop
all the foreign key constraints would do the trick. Maybe that won't be too
difficult to implement?
>
>
>>- no support (as far as I could tell) for generating schema update scripts
>
>
> Yes. Torque is "all or nothing".
So, schema updates would always need hand coded scripts if we stick with Torque if
I understand you correctly.
>
>
>>- no direct execution of ddl/data sql from xml, only through first written out
>> scripts. For execution at runtime from a portal/servlet application this isn't blocking
>> but not exactly what I really would like.
>
>
> Correct. The whole generator has been designed as an additional
> wart^W^Waddition to ant. It definitely needs refactoring and
> personally I want to look into dropping the whole Velocity
> templating/Texen shebang. Texen has given us nothing but headaches
> starting with logging and error reporting and ending up with gobbling
> up memory like mad... Personally, I believe an XSLT solution would be
> nicer but I sorely lack time to do any work on this ATM.
That would mean a quite a rewrite of the generator I think. I wonder if
that would be very profitable in the light of the commons-sql already able
to do this...
>
> Hibernate e.g. has a better solution by generating the creation SQL
> directly at run-time. We still shine at the source code generation for
> the classes/peers though.
>
>
>>- data xml to sql translation is very limited. We at least need proper
>> date and timestamp (in configurable format) value transformation support, as well as
>> support for proper BIT mapping (an issue with PostgreSQL which seems to have
>> been reported several times on the Torque list). Furthermore, I'd like to be able
>> to use variables for values like $now or $yesterday, especially for date/timestamp
>> fields.
>
>
> Support for custom functions would be nice, too (talk encrypting passwords...).
Yes. But if I read you correctly, we shouldn't expect those from Torque soon?
>
>
>>- overriding only a few generator templates (like the table.vm and tablefk.vm for
>> to fix the mysql issue) requires us to maintain *all* templates: torque isn't able
>> to resolve missing templates from the classpath if templates from the file system are to be
>> used.
>
>
> Texen again. I hope to become a velocity committer at some point (it
> was basically agreed upon at ApacheCon with Geir and Daniel but no CfV
> yet. :-( ).
:-(
>
>
>>I've started reviewing commons-sql this morning. Although it is still in the commons sandbox,
>>I think it might be a (far) better match for our requirements.
>
>
> I've read about commons-sql (it will probably move to the DB project
Do you have any more information about that? Probability, timetable?
> (maybe make a short stint through the incubator) soon) yesterday. I
> haven't looked at it yet, but everyone that I hear talking about it,
> really likes it. So yes, I will definitely look at it.
>
>
>>Especially direct ddl/sql execution is provided as well as schema update at runtime (!).
>>Its datatype mapping support also looks far more extensive (including number, date and
>>timestamp format support) than what I've seen in Torque. I expect the PostgreSQL problems
>>Chris Custine has to be far less then we have right now with Torque.
>
>
>>Finally, it also supports (and uses) templating. Although I only see jelly scripts (eek) in the
>>codebase, the documentation also speaks of velocity support.
>
>
> Even JvZ has admitted that introducing Jelly was a mistake. :-)
;-)
>
>
>>I haven't tested it out yet, so it is far too early to conclude it is what I'm looking for,
>>but it sure does look promising.
>
>
>>No disregard to Torque or its team, but considering our requirements and the current features
>>supported by Torque, do you think we should stick with it, or maybe better try commons-sql
>>(or even something of our own which I don't really want if not needed).
>
>
> Sure. Torque _is_ a smallish project and lacks in some aspects (you
> did manage to put your fingers on quite a number of sore spots in the
> generator. :-) ). Bad metadata for the DBs is another thing that I
> really hate.
>
>
>>We hope to put out the final release Jetspeed-2 early next year, and our current problems with
>>the M1 release as well during development need to be addressed soon.
>>If you do think Torque can meet our needs, how much work and time would it cost?
>
>
> To clean up everything that you mentioned above? A lot. I freely
> admit, that for some things that you like to have (update scripts
> e.g.) Torque simply has not support. It was not really intended for
> this. For you immediate needs, I cannot give you an estimate or time
> frame, sorry. I will open a ticket for your points in the TTS so they
> won't be forgotten.
Thanks, I've already seen them.
But, to be frank as well: what would your suggestion be?
Stick with Torque and work through all the issues which will take a lot of time as you said
and might even never be resolved fully, or start investigating a possible migration to commons-sql?
I'm more than willing to support the Torque team in resolving these issues if there is any change
they can be resolved (within time).
Also, the initial response from other Jetspeed team members and users on the suggestion of
David Sean Taylor to try to fix the problems with the Torque team first is +1.
But, my time is limited and I really would like to keep my primary focus on Jetspeed-2 features,
not get lost in DB configuration problems.
So I'm not convinced (yet), certainly not after your frankly given response on the current state of Torque,
we should wait and see.
My impression right now is that we have a much closer match with commons-sql already and that our
near future needs (schema updates, direct ddl/sql execution) will prove that even more so.
Thanks again for your response.
Regards, Ate
>
>
>>Thanks again for your response and I hope we can help each other out with this.
>>I just subscribed to the Torque user and dev lists and also sending this response to the latter.
>
>
> Cool.
>
>
>>If you respond, I'd appreciate it if you would send it to the jetspeed-dev list too.
>
>
> Let's see if cross-posting with this newsreader works out... :-)
>
> Regards
> Henning
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jetspeed-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jetspeed-dev-help@jakarta.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by Ate Douma <at...@douma.nu>.
Henning, thanks for your frankly given response.
Several comments further below.
Henning P. Schmiedehausen wrote:
> Ate Douma <at...@douma.nu> writes:
>
>
>>Henning P. Schmiedehausen wrote:
>
>>In my initial message, http://nagoya.apache.org/eyebrowse/ReadMsg?listId=22&msgNo=19387,
>>I described a few problems we are encountering with Torque right now.
>
>
> Just read it.
>
>>After reviewing the generator and its templates I found the following lacking or missing:
>>- separate generation and/or suppression of drop table statements
>
>
> Yep. the drops should be turn-off-able using a property. They are not.
>
>
>>- order of drop, create table statements causing conflicting result for
>> different database:
>> - for hsql a detail table must be generated (thus dropped) before the
>> parent table otherwise a fk constraint violation is thrown
>> - for mysql (also with innoDB) the foreign keys are generated inline and thus
>> changing the order doesn't help: we need to drop the child table (or constraints)
>> by hand, something Torque doesn't help us with
>
>
> This is difficult to achieve IMHO because the sequence of the tables
> is controlled by the Control.vm template and does not vary for the
> different databases. It _should_ be possible to suppress foreign key
> checking temporarily thus allowing arbitrary creation sequences with
> some databases but I am no real wizard here. This needs more input
> from other developers / users.
I think the suggestion made by Raphaël Luta on the jetspeed list to first drop
all the foreign key constraints would do the trick. Maybe that won't be too
difficult to implement?
>
>
>>- no support (as far as I could tell) for generating schema update scripts
>
>
> Yes. Torque is "all or nothing".
So, schema updates would always need hand coded scripts if we stick with Torque if
I understand you correctly.
>
>
>>- no direct execution of ddl/data sql from xml, only through first written out
>> scripts. For execution at runtime from a portal/servlet application this isn't blocking
>> but not exactly what I really would like.
>
>
> Correct. The whole generator has been designed as an additional
> wart^W^Waddition to ant. It definitely needs refactoring and
> personally I want to look into dropping the whole Velocity
> templating/Texen shebang. Texen has given us nothing but headaches
> starting with logging and error reporting and ending up with gobbling
> up memory like mad... Personally, I believe an XSLT solution would be
> nicer but I sorely lack time to do any work on this ATM.
That would mean a quite a rewrite of the generator I think. I wonder if
that would be very profitable in the light of the commons-sql already able
to do this...
>
> Hibernate e.g. has a better solution by generating the creation SQL
> directly at run-time. We still shine at the source code generation for
> the classes/peers though.
>
>
>>- data xml to sql translation is very limited. We at least need proper
>> date and timestamp (in configurable format) value transformation support, as well as
>> support for proper BIT mapping (an issue with PostgreSQL which seems to have
>> been reported several times on the Torque list). Furthermore, I'd like to be able
>> to use variables for values like $now or $yesterday, especially for date/timestamp
>> fields.
>
>
> Support for custom functions would be nice, too (talk encrypting passwords...).
Yes. But if I read you correctly, we shouldn't expect those from Torque soon?
>
>
>>- overriding only a few generator templates (like the table.vm and tablefk.vm for
>> to fix the mysql issue) requires us to maintain *all* templates: torque isn't able
>> to resolve missing templates from the classpath if templates from the file system are to be
>> used.
>
>
> Texen again. I hope to become a velocity committer at some point (it
> was basically agreed upon at ApacheCon with Geir and Daniel but no CfV
> yet. :-( ).
:-(
>
>
>>I've started reviewing commons-sql this morning. Although it is still in the commons sandbox,
>>I think it might be a (far) better match for our requirements.
>
>
> I've read about commons-sql (it will probably move to the DB project
Do you have any more information about that? Probability, timetable?
> (maybe make a short stint through the incubator) soon) yesterday. I
> haven't looked at it yet, but everyone that I hear talking about it,
> really likes it. So yes, I will definitely look at it.
>
>
>>Especially direct ddl/sql execution is provided as well as schema update at runtime (!).
>>Its datatype mapping support also looks far more extensive (including number, date and
>>timestamp format support) than what I've seen in Torque. I expect the PostgreSQL problems
>>Chris Custine has to be far less then we have right now with Torque.
>
>
>>Finally, it also supports (and uses) templating. Although I only see jelly scripts (eek) in the
>>codebase, the documentation also speaks of velocity support.
>
>
> Even JvZ has admitted that introducing Jelly was a mistake. :-)
;-)
>
>
>>I haven't tested it out yet, so it is far too early to conclude it is what I'm looking for,
>>but it sure does look promising.
>
>
>>No disregard to Torque or its team, but considering our requirements and the current features
>>supported by Torque, do you think we should stick with it, or maybe better try commons-sql
>>(or even something of our own which I don't really want if not needed).
>
>
> Sure. Torque _is_ a smallish project and lacks in some aspects (you
> did manage to put your fingers on quite a number of sore spots in the
> generator. :-) ). Bad metadata for the DBs is another thing that I
> really hate.
>
>
>>We hope to put out the final release Jetspeed-2 early next year, and our current problems with
>>the M1 release as well during development need to be addressed soon.
>>If you do think Torque can meet our needs, how much work and time would it cost?
>
>
> To clean up everything that you mentioned above? A lot. I freely
> admit, that for some things that you like to have (update scripts
> e.g.) Torque simply has not support. It was not really intended for
> this. For you immediate needs, I cannot give you an estimate or time
> frame, sorry. I will open a ticket for your points in the TTS so they
> won't be forgotten.
Thanks, I've already seen them.
But, to be frank as well: what would your suggestion be?
Stick with Torque and work through all the issues which will take a lot of time as you said
and might even never be resolved fully, or start investigating a possible migration to commons-sql?
I'm more than willing to support the Torque team in resolving these issues if there is any change
they can be resolved (within time).
Also, the initial response from other Jetspeed team members and users on the suggestion of
David Sean Taylor to try to fix the problems with the Torque team first is +1.
But, my time is limited and I really would like to keep my primary focus on Jetspeed-2 features,
not get lost in DB configuration problems.
So I'm not convinced (yet), certainly not after your frankly given response on the current state of Torque,
we should wait and see.
My impression right now is that we have a much closer match with commons-sql already and that our
near future needs (schema updates, direct ddl/sql execution) will prove that even more so.
Thanks again for your response.
Regards, Ate
>
>
>>Thanks again for your response and I hope we can help each other out with this.
>>I just subscribed to the Torque user and dev lists and also sending this response to the latter.
>
>
> Cool.
>
>
>>If you respond, I'd appreciate it if you would send it to the jetspeed-dev list too.
>
>
> Let's see if cross-posting with this newsreader works out... :-)
>
> Regards
> Henning
>
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org
Re: [J2] Vote: Torque schema generation problems - how to proceed?
Posted by "Henning P. Schmiedehausen" <hp...@intermeta.de>.
Ate Douma <at...@douma.nu> writes:
>Henning P. Schmiedehausen wrote:
>> Chris Custine <ch...@gmail.com> writes:
>>
>>
>>>I think you may want to consider these items in the context of fixing
>>>the total nightmare of using torque-gen with Postgres as well. I have
>>>posted several times about the hoops I jump through to get JS2 and
>>>Postgres to work together, and your suggestions look like a good
>>>opportunity to fix those problems with torque-gen and Postgres. To be
>>>clear, no single project is at fault (Postgres, Torque, or JS2), but
>>>some minor bad choices in each are making Postgres support really
>>>crappy right now in JS2.
>>
>>
>> The Torque people right over there at torque-dev@db.apache.org would
>> be really happy to know what you consider "total nightmare".
>>
>> (Personally I'm running a PostgreSQL based Project using sequences as
>> ID generators with ~4000 tables in 52 schemas inside a database quite
>> successful. All tables are generated an maintained by Torque and
>> accessed through Torque and Hibernate).
>>
>> (Folks, if you have trouble with Torque and/or improvements or wishes,
>> LET US KNOW!). Not many Torque people scan the lists of <arbitrary
>> project>. Just come over to torque-dev or torque-user and tell us. We
>> are not only willing to listen, we are even willing to act and improve
>> our code to cater the needs of our users. That's what a project is all
>> about.
>Henning, thanks for the response and for scanning our list. Seems we aren't just
>an arbitrary project to you at least ;-)
I scan jetspeed-* because it is built on Turbine (at least 1.5) and I
still would like to find time (or someone that pays for it. ;-) ) to
work more with Jetspeed.
>In my initial message, http://nagoya.apache.org/eyebrowse/ReadMsg?listId=22&msgNo=19387,
>I described a few problems we are encountering with Torque right now.
Just read it.
>As far as I can see, Torque really shines where its Pear support is needed
>(like in Jetspeed-1). But, for our Jetspeed-2 requirements which mainly concern
>mapping an abstract schema definition and inserting initial configuration data,
>the current torque generator features are not really sufficient, nor flexible enough.
Yes. The static peers and the fixed connection of "table -> object"
really hurts (an example: In one of my projects I have a lot of tables
that all have the same structure but describe different things. I'd
really love to define the mapping between object and table only once
and then just say "table a maps to class foo. table b maps to class
foo. table c maps to class foo". I ended up with having all classes
that descibe a, b and c implement the same interface and just work
through the interfaces.
>After reviewing the generator and its templates I found the following lacking or missing:
>- separate generation and/or suppression of drop table statements
Yep. the drops should be turn-off-able using a property. They are not.
>- order of drop, create table statements causing conflicting result for
> different database:
> - for hsql a detail table must be generated (thus dropped) before the
> parent table otherwise a fk constraint violation is thrown
> - for mysql (also with innoDB) the foreign keys are generated inline and thus
> changing the order doesn't help: we need to drop the child table (or constraints)
> by hand, something Torque doesn't help us with
This is difficult to achieve IMHO because the sequence of the tables
is controlled by the Control.vm template and does not vary for the
different databases. It _should_ be possible to suppress foreign key
checking temporarily thus allowing arbitrary creation sequences with
some databases but I am no real wizard here. This needs more input
from other developers / users.
>- no support (as far as I could tell) for generating schema update scripts
Yes. Torque is "all or nothing".
>- no direct execution of ddl/data sql from xml, only through first written out
> scripts. For execution at runtime from a portal/servlet application this isn't blocking
> but not exactly what I really would like.
Correct. The whole generator has been designed as an additional
wart^W^Waddition to ant. It definitely needs refactoring and
personally I want to look into dropping the whole Velocity
templating/Texen shebang. Texen has given us nothing but headaches
starting with logging and error reporting and ending up with gobbling
up memory like mad... Personally, I believe an XSLT solution would be
nicer but I sorely lack time to do any work on this ATM.
Hibernate e.g. has a better solution by generating the creation SQL
directly at run-time. We still shine at the source code generation for
the classes/peers though.
>- data xml to sql translation is very limited. We at least need proper
> date and timestamp (in configurable format) value transformation support, as well as
> support for proper BIT mapping (an issue with PostgreSQL which seems to have
> been reported several times on the Torque list). Furthermore, I'd like to be able
> to use variables for values like $now or $yesterday, especially for date/timestamp
> fields.
Support for custom functions would be nice, too (talk encrypting passwords...).
>- overriding only a few generator templates (like the table.vm and tablefk.vm for
> to fix the mysql issue) requires us to maintain *all* templates: torque isn't able
> to resolve missing templates from the classpath if templates from the file system are to be
> used.
Texen again. I hope to become a velocity committer at some point (it
was basically agreed upon at ApacheCon with Geir and Daniel but no CfV
yet. :-( ).
>I've started reviewing commons-sql this morning. Although it is still in the commons sandbox,
>I think it might be a (far) better match for our requirements.
I've read about commons-sql (it will probably move to the DB project
(maybe make a short stint through the incubator) soon) yesterday. I
haven't looked at it yet, but everyone that I hear talking about it,
really likes it. So yes, I will definitely look at it.
>Especially direct ddl/sql execution is provided as well as schema update at runtime (!).
>Its datatype mapping support also looks far more extensive (including number, date and
>timestamp format support) than what I've seen in Torque. I expect the PostgreSQL problems
>Chris Custine has to be far less then we have right now with Torque.
>Finally, it also supports (and uses) templating. Although I only see jelly scripts (eek) in the
>codebase, the documentation also speaks of velocity support.
Even JvZ has admitted that introducing Jelly was a mistake. :-)
>I haven't tested it out yet, so it is far too early to conclude it is what I'm looking for,
>but it sure does look promising.
>No disregard to Torque or its team, but considering our requirements and the current features
>supported by Torque, do you think we should stick with it, or maybe better try commons-sql
>(or even something of our own which I don't really want if not needed).
Sure. Torque _is_ a smallish project and lacks in some aspects (you
did manage to put your fingers on quite a number of sore spots in the
generator. :-) ). Bad metadata for the DBs is another thing that I
really hate.
>We hope to put out the final release Jetspeed-2 early next year, and our current problems with
>the M1 release as well during development need to be addressed soon.
>If you do think Torque can meet our needs, how much work and time would it cost?
To clean up everything that you mentioned above? A lot. I freely
admit, that for some things that you like to have (update scripts
e.g.) Torque simply has not support. It was not really intended for
this. For you immediate needs, I cannot give you an estimate or time
frame, sorry. I will open a ticket for your points in the TTS so they
won't be forgotten.
>Thanks again for your response and I hope we can help each other out with this.
>I just subscribed to the Torque user and dev lists and also sending this response to the latter.
Cool.
>If you respond, I'd appreciate it if you would send it to the jetspeed-dev list too.
Let's see if cross-posting with this newsreader works out... :-)
Regards
Henning
--
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH
hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/
RedHat Certified Engineer -- Jakarta Turbine Development -- hero for hire
Linux, Java, perl, Solaris -- Consulting, Training, Development
What is more important to you...
[ ] Product Security
or [ ] Quality of Sales and Marketing Support
-- actual question from a Microsoft customer survey
---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org