Extending the scope of the struts validation

[Frangois Rey <fr...@altavista.net>]

You may remember some of my earlier postings about a mapper framework that we use as a validation framework. I now have the support in my company to make it open source. However, I first would prefer to:

1- Invite the community to realize that a validation framework, definitely required in Struts, is only part of a larger requirement.
2- Propose the extension of the scope of the TO DO item for validation so that is does a bit more than just validation but also convert the value and transfer them to where they should end up.

I know I've already approached this topic in my previous postings, but I there hasn't been much follow up on this. However I still believe those that have developed web applications with Struts have been confronted to the same problems we have been trying to solve. So instead of submitting a big zip file that may or may not get interest, I first would like to share the reasons that led us to developing the mapper framework.

In our company, after evaluating a few options, we have chosen Struts as the base framework for developing web applications. We have developed our own simple extensions to Struts in order to provide added support for things like i18n and transaction integrity. Form validation was also a top item in our list of needed extensions, but looking at the Struts 1.1 TO DO list we decided to wait and see what the community would come up with. For one of our customer project, we have looked at the validation framework from David Winterfeldt in order to implement form validation. We were very attracted to that framework because it offered configuration-based validation and also client-side validation. However beyond the simple need to validate form properties, we also needed to do type conversion (text->object) on each field and populate another bean (a command bean) with the converted values. Having a framework that only addressed the validation part meant that we would still need to do a lot of custom coding in order to convert values and populate our command beans. And in addition to this, we also needed to do type conversion and populate presentation  beans when receiving result messages from the backend, which means even more custom coding.

I think by now you should understand and hopefully agree with me that validation is actually only the first part of larger need: once values have been validated, you need to do something with them. Whatever is needed to do, it should not be the responsibility of the ActionForm. The latter should remain a View object that is only a placeholder for user input. Business logic should be separated into another set of objects. Actions seem to be a good place for implementing this business logic. However doing so ties the logic to the web application framework in a way that makes the reuse of this logic outside of a web application difficult. So a good application architect would usually factor this business logic into another set of objects that is not dependent from the web application. Often these objects are actually just sending requests to a backend where the real business logic resides. But whatever these objects are (the real Model, a PreparedStatement, or a proxy to a Model layer implemented on another server), the point is that these are objects which need to be initialized and populated with the user input.

Coming back to our project: we have an EJB backend accessed with a messaging framework: you send command messages (objects), and you get result messages (objects). And of course we wanted to avoid lots of custom code in our actions because it's tedious, not so fun to write, and increases the maintenance burden (more lines of code). So we took the decision to develop the mapper framework. What we now have is used almost all over the place in the application we're developing, for the things I mentioned above: validation, type conversion, and bean population, and for both directions: forms -> command messages and result messages -> presentation beans. The code in our Actions.perform() has decreased dramatically to just a few things. When we get to the Action, the ActionForm.validate() has successfully validated the user input with its mapper. In the Action.perform(), all we do is to ask the same mapper to create the command message and populate it with the converted values, send this object to the backend, ask another mapper to process the result message, and then forward to the JSP.

Now that I have explained why validation is just part of a larger story, one might say that a validation-only framework is a good start anyway. The problem with that approach is that actually validation and conversion can be one and the same thing: if you are able to convert a value into something else, this is at least partly validating the value. Using the SimpleDateFormat class to create a Date object from a String value will fail if the string is not in the proper format and does not contain proper data values. So the point is that we should not dissociate validation from type conversion. In some cases the validation is completely fulfilled by a successful conversion, in other cases there is no need to convert but only to validate, and in some other there is no need to validate or convert.
By looking at recent posting I think already most of us agree with the fact that the validation framework should also be able to convert values. I don't think I really need to convince people on this. The real extension that I want to propose to the current scope for validation is the ability to transfer the converted values to their real destination: most likely in a property of another object. Think about it: how much more is it? If the validation and conversion are automated, no custom coding is required for this anymore. But then in the current scope you still need to extract those validated/converted values and, as I explained before, use them to populate other objects. Automating the population of other objects with converted values is not such a big effort when using PropertyUtils, we just have to specify the destination object and the recieving property names at the same place where we specify the validation/conversion. If some of you still think that just a validation/conversion framework is again enough as a starting point, then think about it again: how much more is it have the transfer functionality compared to the lines of custom code you'll have to write to populate other objects with the converted values?

Having a validate/convert/transfer framework is really what we have tried to achieve with the mapper framework. Because it does the job quite well and has increased our productivity, I felt more compelled to write our story rather than just sending a big zip file with the source. Whatever the Struts validation become, I think a framework similar in scope to ours would really benefit to those developing applications with Struts, and would make the Struts offering even more attractive.

If with this posing I also created an interest in making our mapper framework open source, then great. I started to develop it 6 weeks ago and there are plenty of room for improvements. For example it does not generate client-side javascript, it is still creating lots of object at run time (no object recycling), and it only works from one single configuration file (a bottleneck when using a repository within a large team). So making it open source could be a great way to improve it and could give a kick start for the Struts validation framework.

If you want to know more about the mapper, I invite you to read some of my earlier postings, in which I also sent a sample configuration file. The mapper framework has been developed independently from the Struts MVC classes. It is only dependent on these utility packages: 
    - Struts PropertyUtils
    - Struts Digester (and therefore an XML parser - Xerces of course :)
    - The Regexp package from Jakarta (used in one validator )
    - JavaCC/JJTree for parsing custom validation rules

Some of the logic for digesting the XML config file required some extensions to the PropertyUtils and Digester. The PropertyUtils extension is for supporting what we have called 'string keyed' properties, which are properties with accessors of the form getProperty(String key) and setProperty(String key, Object value). The extensions to the digester were for supporting the resolution of system entities (only public entities can be resolved at present), and for adding convenience methods in order to add our own rules to digester. The new rules we use are called SetTopNow, SetNextNow, SetPropertyFromAttribute, SetInitFromObjects, CallMethodNow, etc (hopefully these name gives you a rough indication of what they do).

If the interest is there, I can submit the code (although I'd prefer waiting another week to finish some doc). But I don't think it should be within the Struts project. I also don't think the strutsx would be an ideal place either because as I said it does not depend on Struts, only a bit now because the PropertyUtils and Digester are still part of Struts. I'm more thinking of the Jarakta commons since it is more generic in nature and is has been built on top of things that will be transferred into the Jakarta commons. The other reason why it is better placed in the commons is because it is actually a generic validate/convert/transfer framework which can also have an interesting value for other types of applications, not only for web developement. For example it could be used in a file handler in order to validate each record and produce a message object so be sent to a backend. With the ability to generate code out of the configuration file, performance would be improved by avoiding reflection and could be used for server applications with higher performance requirements. Again lots of things an open source community can do...

François Rey
mailto:francois.rey@capco.com

-- 

_______________________________________________
Get your free email from http://mymail.altavista.com

Re: Extending the scope of the struts validation

[Ron Smith <ro...@rpsenterprises.com>]

That's great that your management supports open sourcing this code.
Personally, I'd be very interested in seeing it.

I had an immediate need for a transformations (data conversion) package
(e.g. Date<->String, ArrayList -> Sorted ArrayList, ...), so I started
work on one recently.
I'm putting the finishing touches on the first (very simple) version.

I completely agree that validation and data conversion (what I'm calling
transformation) should be tied together at a higher level.
I see transformation as a separate package that is not dependent on
validation, but could be used by validation.
In some cases, the act of doing the transformation may be all the
validation that is needed (e.g. if we could convert the String to
an Integer, that's good enough).  I could see scenarios where
you'd want to do some validation before transformation (e.g.
by pattern matching on the String), and some cases where you'd
want to do validation after transformation (e.g. making sure
an integer that's been converted is within some range).

I need to get this transformations package finished up by Monday, so
I'll finish up this first revision and post it to this list.

I'm hoping all of these efforts in the validation/mapping/data
conversion
space can be consolidated soon.



Frangois Rey wrote:
> 
> You may remember some of my earlier postings about a mapper framework that we use as a validation framework. I now have the support in my company to make it open source. However, I first would prefer to:
> 
> ...
> ...
>
> François Rey
> mailto:francois.rey@capco.com
> 
> --
> 
> _______________________________________________
> Get your free email from http://mymail.altavista.com

Re: Extending the scope of the struts validation

[Oleg V Alexeev <go...@penza.net>]

Hello Frangois,

+1 to merge validation and transformation processing.
I like idea that core java transformation classes (SimpleDateFormat
for example) can help to validate inputs in conversion processing
moment. It all already handle locale to transform - i18n supported by
the way.

It think that validation and transformation splitting is way to the
overhead. One step - validation over all values, next step -
transformation for all non String targets. SimpleDateFormat is a good
sample, for my mind, of right way to perform validation &
transformation in plane. It take String, convert to the target type
and throw exception in case of any errors - really, bad input is
exception in conversion process.


Saturday, June 23, 2001, 6:54:49 PM, you wrote:

FR> You may remember some of my earlier postings about a mapper framework that we use as a validation framework. I now have the support in my company to make it open source. However, I first would
FR> prefer to:

FR> 1- Invite the community to realize that a validation framework, definitely required in Struts, is only part of a larger requirement.
FR> 2- Propose the extension of the scope of the TO DO item for validation so that is does a bit more than just validation but also convert the value and transfer them to where they should end up.

FR> I know I've already approached this topic in my previous postings, but I there hasn't been much follow up on this. However I still believe those that have developed web applications with Struts
FR> have been confronted to the same problems we have been trying to solve. So instead of submitting a big zip file that may or may not get interest, I first would like to share the reasons that led
FR> us to developing the mapper framework.

FR> In our company, after evaluating a few options, we have chosen Struts as the base framework for developing web applications. We have developed our own simple extensions to Struts in order to
FR> provide added support for things like i18n and transaction integrity. Form validation was also a top item in our list of needed extensions, but looking at the Struts 1.1 TO DO list we decided to
FR> wait and see what the community would come up with. For one of our customer project, we have looked at the validation framework from David Winterfeldt in order to implement form validation. We
FR> were very attracted to that framework because it offered configuration-based validation and also client-side validation. However beyond the simple need to validate form properties, we also needed
FR> to do type conversion (text->object) on each field and populate another bean (a command bean) with the converted values. Having a framework that only addressed the validation part meant that we
FR> would still need to do a lot of custom coding in order to convert values and populate our command beans. And in addition to this, we also needed to do type conversion and populate presentation 
FR> beans when receiving result messages from the backend, which means even more custom coding.

FR> I think by now you should understand and hopefully agree with me that validation is actually only the first part of larger need: once values have been validated, you need to do something with
FR> them. Whatever is needed to do, it should not be the responsibility of the ActionForm. The latter should remain a View object that is only a placeholder for user input. Business logic should be
FR> separated into another set of objects. Actions seem to be a good place for implementing this business logic. However doing so ties the logic to the web application framework in a way that makes
FR> the reuse of this logic outside of a web application difficult. So a good application architect would usually factor this business logic into another set of objects that is not dependent from the
FR> web application. Often these objects are actually just sending requests to a backend where the real business logic resides. But whatever these objects are (the real Model, a PreparedStatement, or
FR> a proxy to a Model layer implemented on another server), the point is that these are objects which need to be initialized and populated with the user input.

FR> Coming back to our project: we have an EJB backend accessed with a messaging framework: you send command messages (objects), and you get result messages (objects). And of course we wanted to
FR> avoid lots of custom code in our actions because it's tedious, not so fun to write, and increases the maintenance burden (more lines of code). So we took the decision to develop the mapper
FR> framework. What we now have is used almost all over the place in the application we're developing, for the things I mentioned above: validation, type conversion, and bean population, and for both
FR> directions: forms -> command messages and result messages -> presentation beans. The code in our Actions.perform() has decreased dramatically to just a few things. When we get to the Action, the
FR> ActionForm.validate() has successfully validated the user input with its mapper. In the Action.perform(), all we do is to ask the same mapper to create the command message and populate it with
FR> the converted values, send this object to the backend, ask another mapper to process the result message, and then forward to the JSP.

FR> Now that I have explained why validation is just part of a larger story, one might say that a validation-only framework is a good start anyway. The problem with that approach is that actually
FR> validation and conversion can be one and the same thing: if you are able to convert a value into something else, this is at least partly validating the value. Using the SimpleDateFormat class to
FR> create a Date object from a String value will fail if the string is not in the proper format and does not contain proper data values. So the point is that we should not dissociate validation from
FR> type conversion. In some cases the validation is completely fulfilled by a successful conversion, in other cases there is no need to convert but only to validate, and in some other there is no
FR> need to validate or convert.
FR> By looking at recent posting I think already most of us agree with the fact that the validation framework should also be able to convert values. I don't think I really need to convince people on
FR> this. The real extension that I want to propose to the current scope for validation is the ability to transfer the converted values to their real destination: most likely in a property of another
FR> object. Think about it: how much more is it? If the validation and conversion are automated, no custom coding is required for this anymore. But then in the current scope you still need to extract
FR> those validated/converted values and, as I explained before, use them to populate other objects. Automating the population of other objects with converted values is not such a big effort when
FR> using PropertyUtils, we just have to specify the destination object and the recieving property names at the same place where we specify the validation/conversion. If some of you still think that
FR> just a validation/conversion framework is again enough as a starting point, then think about it again: how much more is it have the transfer functionality compared to the lines of custom code
FR> you'll have to write to populate other objects with the converted values?

FR> Having a validate/convert/transfer framework is really what we have tried to achieve with the mapper framework. Because it does the job quite well and has increased our productivity, I felt more
FR> compelled to write our story rather than just sending a big zip file with the source. Whatever the Struts validation become, I think a framework similar in scope to ours would really benefit to
FR> those developing applications with Struts, and would make the Struts offering even more attractive.

FR> If with this posing I also created an interest in making our mapper framework open source, then great. I started to develop it 6 weeks ago and there are plenty of room for improvements. For
FR> example it does not generate client-side javascript, it is still creating lots of object at run time (no object recycling), and it only works from one single configuration file (a bottleneck when
FR> using a repository within a large team). So making it open source could be a great way to improve it and could give a kick start for the Struts validation framework.

FR> If you want to know more about the mapper, I invite you to read some of my earlier postings, in which I also sent a sample configuration file. The mapper framework has been developed
FR> independently from the Struts MVC classes. It is only dependent on these utility packages: 
FR>     - Struts PropertyUtils
FR>     - Struts Digester (and therefore an XML parser - Xerces of course :)
FR>     - The Regexp package from Jakarta (used in one validator )
FR>     - JavaCC/JJTree for parsing custom validation rules

FR> Some of the logic for digesting the XML config file required some extensions to the PropertyUtils and Digester. The PropertyUtils extension is for supporting what we have called 'string keyed'
FR> properties, which are properties with accessors of the form getProperty(String key) and setProperty(String key, Object value). The extensions to the digester were for supporting the resolution of
FR> system entities (only public entities can be resolved at present), and for adding convenience methods in order to add our own rules to digester. The new rules we use are called SetTopNow,
FR> SetNextNow, SetPropertyFromAttribute, SetInitFromObjects, CallMethodNow, etc (hopefully these name gives you a rough indication of what they do).

FR> If the interest is there, I can submit the code (although I'd prefer waiting another week to finish some doc). But I don't think it should be within the Struts project. I also don't think the
FR> strutsx would be an ideal place either because as I said it does not depend on Struts, only a bit now because the PropertyUtils and Digester are still part of Struts. I'm more thinking of the
FR> Jarakta commons since it is more generic in nature and is has been built on top of things that will be transferred into the Jakarta commons. The other reason why it is better placed in the
FR> commons is because it is actually a generic validate/convert/transfer framework which can also have an interesting value for other types of applications, not only for web developement. For
FR> example it could be used in a file handler in order to validate each record and produce a message object so be sent to a backend. With the ability to generate code out of the configuration file,
FR> performance would be improved by avoiding reflection and could be used for server applications with higher performance requirements. Again lots of things an open source community can do...

FR> François Rey
FR> mailto:francois.rey@capco.com




-- 
Best regards,
 Oleg                            mailto:gonza@penza.net

Re: Extending the scope of the struts validation

[David Winterfeldt <dw...@yahoo.com>]

I'm intersted in seeing the code too.  If my
validation framework does more, maybe we can make them
work in tandem.  

I've actually been working on making the validation
framework work independant of Struts.  A friend of
mine wants to be able to configure a different set of
rules just to run on an EJB layer.  The work is going
well.  I need to test it a little bit, but I'm hoping
to post the update by the end of the weekend.

David

--- Ted Husted <hu...@apache.org> wrote:
> Personally, I'm probably +1 on all of this, though
> it's hard to be sure
> until the code is on the table ;-)
> 
> A very good approach, regardless of whether you want
> to offer it to
> Struts or to the Commons, is to setup a portal page
> someplace, as David,
> Oleg, and Cedric have done. You could always post
> what you have now, and
> update the rest as soon as you are able. (Release
> early, release often
> ;-)
> 
> It's very good news that you were able to persuade
> your company to make
> this public. I believe a validation framework is
> something the Java
> community can really use right now.
> 
> I also like the stance that this is something that
> we can use in Struts
> AND ELSEWHERE, since a focal point of a framework
> like Struts should be
> to allow as much code as possible to be shared with
> other environments.
> 
> Frangois Rey wrote:
> > If the interest is there, I can submit the code
> (although I'd prefer waiting another week to finish
> some doc). But I don't think it should be within the
> Struts project. I also don't think the strutsx would
> be an ideal place either because as I said it does
> not depend on Struts, only a bit now because the
> PropertyUtils and Digester are still part of Struts.
> I'm more thinking of the Jarakta commons since it is
> more generic in nature and is has been built on top
> of things that will be transferred into the Jakarta
> commons. The other reason why it is better placed in
> the commons is because it is actually a generic
> validate/convert/transfer framework which can also
> have an interesting value for other types of
> applications, not only for web developement. For
> example it could be used in a file handler in order
> to validate each record and produce a message object
> so be sent to a backend. With the ability to
> generate code out of the configuration file,
> performance would be improved !
> by  avoiding reflection and could be used for server
> applications with higher performance requirements.
> Again lots of things an open source community can
do...


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

Re: Extending the scope of the struts validation

[Ted Husted <hu...@apache.org>]

Personally, I'm probably +1 on all of this, though it's hard to be sure
until the code is on the table ;-)

A very good approach, regardless of whether you want to offer it to
Struts or to the Commons, is to setup a portal page someplace, as David,
Oleg, and Cedric have done. You could always post what you have now, and
update the rest as soon as you are able. (Release early, release often
;-)

It's very good news that you were able to persuade your company to make
this public. I believe a validation framework is something the Java
community can really use right now.

I also like the stance that this is something that we can use in Struts
AND ELSEWHERE, since a focal point of a framework like Struts should be
to allow as much code as possible to be shared with other environments.

Frangois Rey wrote:
> If the interest is there, I can submit the code (although I'd prefer waiting another week to finish some doc). But I don't think it should be within the Struts project. I also don't think the strutsx would be an ideal place either because as I said it does not depend on Struts, only a bit now because the PropertyUtils and Digester are still part of Struts. I'm more thinking of the Jarakta commons since it is more generic in nature and is has been built on top of things that will be transferred into the Jakarta commons. The other reason why it is better placed in the commons is because it is actually a generic validate/convert/transfer framework which can also have an interesting value for other types of applications, not only for web developement. For example it could be used in a file handler in order to validate each record and produce a message object so be sent to a backend. With the ability to generate code out of the configuration file, performance would be improved !
by  avoiding reflection and could be used for server applications with higher performance requirements. Again lots of things an open source community can do...

Re: Extending the scope of the struts validation

[William Jaynes <wj...@mediaone.net>]

One of the things I like in your description is the addition of your 'string
keyed' properties. These sound like dynamic properties. We are going to need
dynamic properties as part of ActionForms if we are going to implement dynamic
forms. The problem I've had with the validation discussion involving JavaBean
contrained properties is that a JavaBean isn't dynamic enough to handle dynamic
forms. If I don't know what properties my form will have until runtime, how can
I use a JavaBean?

Just for a different point of view it is instructive to look at forms and
validation in the Barracuda world. Their forms are basically maps (all
properties are 'string keyed'), and any number of Validators can be attached to
each property (for individual validation) and each form (for overall
validation). One can subclass their Form class, as we do with ActionForm.. But
one can also create a form at runtime, adding properties (or FormElements as
they call them) on the fly (for example, from an XML description).

So I like your ideas because they sound more flexible and dynamic than what
Struts currently has, and more flexible than what a straight JavaBean framework
would allow.

Will

> If you want to know more about the mapper, I invite you to read some of my
earlier postings, in which I also sent a sample configuration file. The mapper
framework has been developed independently from the Struts MVC classes. It is
only dependent on these utility packages:
>     - Struts PropertyUtils
>     - Struts Digester (and therefore an XML parser - Xerces of course :)
>     - The Regexp package from Jakarta (used in one validator )
>     - JavaCC/JJTree for parsing custom validation rules
>
> Some of the logic for digesting the XML config file required some extensions
to the PropertyUtils and Digester. The PropertyUtils extension is for supporting
what we have called 'string keyed' properties, which are properties with
accessors of the form getProperty(String key) and setProperty(String key, Object
value). The extensions to the digester were for supporting the resolution of
system entities (only public entities can be resolved at present), and for
adding convenience methods in order to add our own rules to digester. The new
rules we use are called SetTopNow, SetNextNow, SetPropertyFromAttribute,
SetInitFromObjects, CallMethodNow, etc (hopefully these name gives you a rough
indication of what they do).