Syntax change for directives

[Jason van Zyl <jv...@periapt.com>]

Hi,

I've been playing around with directives for the
last day, thinking about how pluggable directives
would work best and what some of the current
problems are.

Brian Goetz and I have been shooting a couple
emails back and forth regarding the subject
and he has found that using a JAVACODE
production solves the problem for WM. A JAVACODE
production is basically a mini hand written
parser in the midst of a generated JavaCC
parser. The need for a JAVACODE production
arises from contextual ambiguity. For example:

#foreach $element in $list $element #end
            1      2    3     4

If all directives are pluggable then there
is nothing specific in the grammar to
catch a #foreach $element in $list. You are basically
looking for #anything, then a list of arguments.
In the above example it's not possible to tell
that (4) is in the body of the #foreach without
giving the parser a hint i.e. the number of
arguments to the directive. So a #foreach has
three arguments so what follows must be the
body of loop.

Now what Brian has come up with is really
quite ingenious! But I believe there is a
simpler solution. The JAVACODE production is
required by the mere presence of contextual
ambiguity. So I say remove the contextual
ambiguity and use a much simpler mechanism,
removing the need for a JAVACODE production.

I've had discussions with Terence about this, 
having well defined entry/exit
points for certain parsing operations. For example
JavaDoc has the very simple /** */ entry/exit
markers. And in the chat room last night Geir
suggested the same type of system for pluggable
directives.

So to remove the contextual ambiguity, and the
requirement of a JAVACODE production I would say
that delimiting the arguments for a directive
would be much simpler and actually be more
flexible:

#foreach ($element in $list) $element #end

What this would also allow is a variable
number of arguments to the directives. Here's
hypothetical examples of where a variable number
of directives might be useful.

#set $greeting = "Hi $name! Well you know $today is your lucky day!"

#expand ($greeting with $a $b)

But you could put as many parameters in $greeting as you
liked, and you could expand the greeting with the list
of values to match those parameters. Maybe a contrived
example and useless but I'm sure a variable number
of arguments to a directive might be useful.

The #set directive would be an exception, it would
look as it does now. But even single argument directives
would have to be enclosed in the delimiters. So we
would have things like the following:

#foreach ($element in $list)
    $element
#end

#expand ($greeting with $a $b)

#escape ( $a ) [ not in VM just using a WM example]

But directives with no arguments could probably
just be left as:

#compress
   text to
   compress
#end

I wanted to throw the idea out and see what people
thought. I think it would make things a great
deal simpler WRT pluggable directives and a lot
more flexible. You could decide to have optional
parameters to a directive, and you could look
for the number of directive arguments passed to
the directive then decide how to proceed.

If people are in agreement then I would like to
make the changes ASAP and alter the testbed 
and give it a whirl!

Comments?

jvz.

-- 

Jason van Zyl
jvanzyl@periapt.com

Re: Syntax change for directives

[Jason van Zyl <jv...@periapt.com>]

On Sun, 1 Oct 2000, Andy Armstrong wrote:

> First a short introduction: I've just joined the list today and started
> having a look at Velocity mainly because in a lot of ways it's close to
> a servlet based template engine (Takhini) that we've been developing in
> house (with a view to O/S release). I'll post a brief outline of that
> separately -- I'm hopeful that some of the work we've done may be useful
> here, and I'm quite prepared -- if we can see a way to migrate our
> existing Takhini based systems to Velocity -- to unfork and merge any
> useful bits of Takhini with Velocity.

That would be fantastic!
 
> I hope regulars here will forgive me for venturing an opinion on this
> subject. Normally I'd lurk for a while, but because we've covered this
> ground with Takhini I'm hopeful that I might have something to
> contribute immediately. Please feel free to put me in my place if I
> overstep the mark.

Oh, no worries :)

> Anyway, the first thing that occurs to me is that while having all
> directives pluggable seems in some ways very cool, I suspect that in
> practice once you've provided a full range of control structures (if,
> for, while, repeat, case, whatever) you're unlikely to want to plug in
> any additional directives with anything other than procedure call
> semantics (i.e. a name followed by some arguments). Pluggable syntax for
> control structures is a whole can of worms if you're going to have
> things like matching opening and closing keywords (#foreach .. #end)
> because you have to have pluggable syntax checking for them, including
> support for nested directives.

There are some directives that I would definitely like to leave
in the JavaCC grammar. The #set and #if/#elseif/#else directives
for example. But the #foreach directive works just fine as
a pluggable direcitive.

> Our approach with Takhini has been to hardwire the control structures
> (and things like assignments) into the language (we only currently have
> if/else and loop structures) and also make expression syntax hardwired,
> but make it possible for user code to extend the language by exposing
> functions (variadic) and variables/constants.

Some of the control structures are hard-wired in Velocity, all
the expression syntax as well.

> Next, a word about loops. Here's a little fragment of Takhini (Takhini
> substitutions are marked with '%'):
> 
>  %.name=['Fred','Jim','Albert']%   <!-- a list of names -->
>  %.age=[36,67,82]%                 <!-- a list of ages  -->
>  <table>
>     %{%            <!-- start of loop -->
>         <tr><td>%name%</td><td>%age%</td></tr>
>     %}%            <!-- end of loop -->
>  </table>
> 
> which translates into
> 
>  <table>
> 
>         <tr><td>Fred</td><td>36</td></tr>
>         <tr><td>Jim</td><td>67</td></tr>
>         <tr><td>Albert</td><td>82</td></tr>
> 
>  </table>
> 
> To make sense of that you need a little background. Firstly list values
> are vaguely magical in Takhini; for example most of the operators do
> special things with lists, so
> 
>   1 + 2                --> 3
>   "Hello, " + "World"  --> "Hello, World"
>   "Hello, " + ["World", "Chums"] --> ["Hello, World", "Hello, Chums"]
> 
> and loops have a special understanding of list objects, viz a loop will
> iterate once for each element in the largest list that is evaluated
> inside it, so the act of evaluating |name| and |age| inside the loop
> makes it iterate three times (if one list had four elements an empty
> string would have been provided for the missing element in the smaller
> string).
> 
> While at first glance this might look a little brain dead it actually
> makes a lot of sense in practice. One of the traditional problems with
> foreach style loops is that you can't iterate over the corresponding
> elements in multiple lists at one time. Also with foreach loops you
> don't typically have access to the loop index. In Takhini we have a 
> special variable ('$') that represents the current loop index. Enclosing
> loop indexes are $$ (for the innermost enclosing loop), $$$ for the one
> outside that and so on.

We have also discussed throwing in a special variable to
keep track of the loop index. Very easy to add.

> I'm going to post some more background about Takhini separately --
> hopefully we've got some bits and pieces that may be of interest.

Great! Send it on down the pipe :)

jvz.

-- 

Jason van Zyl
jvanzyl@periapt.com

Re: Syntax change for directives

[Andy Armstrong <an...@tagish.com>]

First a short introduction: I've just joined the list today and started
having a look at Velocity mainly because in a lot of ways it's close to
a servlet based template engine (Takhini) that we've been developing in
house (with a view to O/S release). I'll post a brief outline of that
separately -- I'm hopeful that some of the work we've done may be useful
here, and I'm quite prepared -- if we can see a way to migrate our
existing Takhini based systems to Velocity -- to unfork and merge any
useful bits of Takhini with Velocity.

I hope regulars here will forgive me for venturing an opinion on this
subject. Normally I'd lurk for a while, but because we've covered this
ground with Takhini I'm hopeful that I might have something to
contribute immediately. Please feel free to put me in my place if I
overstep the mark.

Anyway, the first thing that occurs to me is that while having all
directives pluggable seems in some ways very cool, I suspect that in
practice once you've provided a full range of control structures (if,
for, while, repeat, case, whatever) you're unlikely to want to plug in
any additional directives with anything other than procedure call
semantics (i.e. a name followed by some arguments). Pluggable syntax for
control structures is a whole can of worms if you're going to have
things like matching opening and closing keywords (#foreach .. #end)
because you have to have pluggable syntax checking for them, including
support for nested directives.

Our approach with Takhini has been to hardwire the control structures
(and things like assignments) into the language (we only currently have
if/else and loop structures) and also make expression syntax hardwired,
but make it possible for user code to extend the language by exposing
functions (variadic) and variables/constants.

Next, a word about loops. Here's a little fragment of Takhini (Takhini
substitutions are marked with '%'):

 %.name=['Fred','Jim','Albert']%   <!-- a list of names -->
 %.age=[36,67,82]%                 <!-- a list of ages  -->
 <table>
    %{%            <!-- start of loop -->
        <tr><td>%name%</td><td>%age%</td></tr>
    %}%            <!-- end of loop -->
 </table>

which translates into

 <table>

        <tr><td>Fred</td><td>36</td></tr>
        <tr><td>Jim</td><td>67</td></tr>
        <tr><td>Albert</td><td>82</td></tr>

 </table>

To make sense of that you need a little background. Firstly list values
are vaguely magical in Takhini; for example most of the operators do
special things with lists, so

  1 + 2                --> 3
  "Hello, " + "World"  --> "Hello, World"
  "Hello, " + ["World", "Chums"] --> ["Hello, World", "Hello, Chums"]

and loops have a special understanding of list objects, viz a loop will
iterate once for each element in the largest list that is evaluated
inside it, so the act of evaluating |name| and |age| inside the loop
makes it iterate three times (if one list had four elements an empty
string would have been provided for the missing element in the smaller
string).

While at first glance this might look a little brain dead it actually
makes a lot of sense in practice. One of the traditional problems with
foreach style loops is that you can't iterate over the corresponding
elements in multiple lists at one time. Also with foreach loops you
don't typically have access to the loop index. In Takhini we have a 
special variable ('$') that represents the current loop index. Enclosing
loop indexes are $$ (for the innermost enclosing loop), $$$ for the one
outside that and so on.

I'm going to post some more background about Takhini separately --
hopefully we've got some bits and pieces that may be of interest.

Jason van Zyl wrote:
> 
> Hi,
> 
> I've been playing around with directives for the
> last day, thinking about how pluggable directives
> would work best and what some of the current
> problems are.
> 
> Brian Goetz and I have been shooting a couple
> emails back and forth regarding the subject
> and he has found that using a JAVACODE
> production solves the problem for WM. A JAVACODE
> production is basically a mini hand written
> parser in the midst of a generated JavaCC
> parser. The need for a JAVACODE production
> arises from contextual ambiguity. For example:
> 
> #foreach $element in $list $element #end
>             1      2    3     4
> 
> If all directives are pluggable then there
> is nothing specific in the grammar to
> catch a #foreach $element in $list. You are basically
> looking for #anything, then a list of arguments.
> In the above example it's not possible to tell
> that (4) is in the body of the #foreach without
> giving the parser a hint i.e. the number of
> arguments to the directive. So a #foreach has
> three arguments so what follows must be the
> body of loop.
> 
> Now what Brian has come up with is really
> quite ingenious! But I believe there is a
> simpler solution. The JAVACODE production is
> required by the mere presence of contextual
> ambiguity. So I say remove the contextual
> ambiguity and use a much simpler mechanism,
> removing the need for a JAVACODE production.
> 
> I've had discussions with Terence about this,
> having well defined entry/exit
> points for certain parsing operations. For example
> JavaDoc has the very simple /** */ entry/exit
> markers. And in the chat room last night Geir
> suggested the same type of system for pluggable
> directives.
> 
> So to remove the contextual ambiguity, and the
> requirement of a JAVACODE production I would say
> that delimiting the arguments for a directive
> would be much simpler and actually be more
> flexible:
> 
> #foreach ($element in $list) $element #end
> 
> What this would also allow is a variable
> number of arguments to the directives. Here's
> hypothetical examples of where a variable number
> of directives might be useful.
> 
> #set $greeting = "Hi $name! Well you know $today is your lucky day!"
> 
> #expand ($greeting with $a $b)
> 
> But you could put as many parameters in $greeting as you
> liked, and you could expand the greeting with the list
> of values to match those parameters. Maybe a contrived
> example and useless but I'm sure a variable number
> of arguments to a directive might be useful.
> 
> The #set directive would be an exception, it would
> look as it does now. But even single argument directives
> would have to be enclosed in the delimiters. So we
> would have things like the following:
> 
> #foreach ($element in $list)
>     $element
> #end
> 
> #expand ($greeting with $a $b)
> 
> #escape ( $a ) [ not in VM just using a WM example]
> 
> But directives with no arguments could probably
> just be left as:
> 
> #compress
>    text to
>    compress
> #end
> 
> I wanted to throw the idea out and see what people
> thought. I think it would make things a great
> deal simpler WRT pluggable directives and a lot
> more flexible. You could decide to have optional
> parameters to a directive, and you could look
> for the number of directive arguments passed to
> the directive then decide how to proceed.
> 
> If people are in agreement then I would like to
> make the changes ASAP and alter the testbed
> and give it a whirl!
> 
> Comments?
> 
> jvz.
> 
> --
> 
> Jason van Zyl
> jvanzyl@periapt.com

-- 
Andy Armstrong, Tagish

Re: Syntax change for directives

["Geir Magnusson Jr." <ge...@optonline.net>]

needless to say, +1

geir

-- 
Geir Magnusson Jr.                                    
geirm@optonline.com
"Congress shall make no law abridging ...  the freedom of sXXXch, ... or
the
right of the people peaceably to XXXemble, and to peXXXion the
government for 
a redress of grievances." -even the Founding Fathers needed content
filters

Re: Syntax change for directives

[bob <bo...@accurev.com>]

> #foreach ($element in $list) $element #end
> #expand ($greeting with $a $b)

Being a parsing-guy m'self, I understand the problems
associated with directives, and would like to vote

	+1

on using parens for directives.  I see it as increasing
flexibility, performance, and possibly lowering time-to-market.

Re: Syntax change for directives

[Jason van Zyl <jv...@periapt.com>]

On Tue, 10 Oct 2000, Brian Goetz wrote:

> 
> >#foreach $element in $list $element #end
> >             1      2    3     4
> >
> >If all directives are pluggable then there
> >is nothing specific in the grammar to
> >catch a #foreach $element in $list. You are basically
> >looking for #anything, then a list of arguments.
> >In the above example it's not possible to tell
> >that (4) is in the body of the #foreach without
> >giving the parser a hint i.e. the number of
> >arguments to the directive. So a #foreach has
> >three arguments so what follows must be the
> >body of loop.
> 
> Actually, what I did in WM is a little more complicated, but this is 
> close.  Each directive has a directive descriptor, which includes a list of 
> argument descriptors.  Argument descriptors describe an argument, and there 
> are defined types for Conditional, LValue, RValue, Keyword, Block, etc.  So 
> the argument list for #foreach would be: ( LValue, Keyword("in"), RValue, 
> Block )
> 
> While this solves the problem of backward compatibility in WM, it does make 
> the parser more complicated.  Jason's idea of delimiting the argument list 
> with ( .. ) seems to work, although you're still going to want to be able 
> to say that some directives require a block (like #if and #foreach) and 
> some don't (like #set.)  I think this will still require some, but fewer, 
> JAVACODE productions.

You still have to know whether the directive has blocks, yes, but we
have no JAVACODE productions and everything is working
fine. Geir and I are just finishing what we hope will be the
first production quality parser.

jvz.

Re: Syntax change for directives

[Brian Goetz <br...@quiotix.com>]

>#foreach $element in $list $element #end
>             1      2    3     4
>
>If all directives are pluggable then there
>is nothing specific in the grammar to
>catch a #foreach $element in $list. You are basically
>looking for #anything, then a list of arguments.
>In the above example it's not possible to tell
>that (4) is in the body of the #foreach without
>giving the parser a hint i.e. the number of
>arguments to the directive. So a #foreach has
>three arguments so what follows must be the
>body of loop.

Actually, what I did in WM is a little more complicated, but this is 
close.  Each directive has a directive descriptor, which includes a list of 
argument descriptors.  Argument descriptors describe an argument, and there 
are defined types for Conditional, LValue, RValue, Keyword, Block, etc.  So 
the argument list for #foreach would be: ( LValue, Keyword("in"), RValue, 
Block )

While this solves the problem of backward compatibility in WM, it does make 
the parser more complicated.  Jason's idea of delimiting the argument list 
with ( .. ) seems to work, although you're still going to want to be able 
to say that some directives require a block (like #if and #foreach) and 
some don't (like #set.)  I think this will still require some, but fewer, 
JAVACODE productions.



--
Brian Goetz
Quiotix Corporation
brian@quiotix.com           Tel: 650-843-1300            Fax: 650-324-8032

http://www.quiotix.com