You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@velocity.apache.org by Jason Pettiss <ja...@CatalisHealth.com> on 2005/07/06 16:37:41 UTC
Escapage of escapage
I'm going insane here, maybe someone familiar with the parsing of
velocity templates can help out. I have a document which is all
'schmoo'-- it's not a velocity template... yet. I need to make sure
this document, when parsed by velocity, comes out exactly like it is--
so that all schmoo is schmoo. But I'm having a hell of a time doing this.
I figured that it would be a simple matter of escaping every backslash
in the document with another backslash, and then escaping every $ and #
with a backslash. But this doubles the backslashes and does random
stuff with the dollar signs (the directives, at least, all have their
wings clipped).
In particular I notice that \$foo is not guaranteed to come out as $foo
like I would expect. After all what good is an escape sequence if it
doesn't actually escape out of the parsing behavior! But I find that
depending on whether 'foo' happens to be set or not, \$foo will either
come out $foo or \$foo. This extra backslash is totally uncalled for--
I have no way to escape it, and I have no way of knowing whether it will
appear or not since I don't actually care to know what's in the document
I'm handling. I mean, maybe it was legitimately talking about the cost
of foo, or something.
For those curious, I'm actually converting a gigantic content repository
from an ancient templating language into Velocity. It involves taking a
document, escaping everything that Velocity might think is Velocity
because it's just schmoo, and then sprinkling Velocity directives and
variables all over the document (this IS Velocity). How do I tell
Velocity to piss off without having backslashes randomly appear all over
the place, and without actually having to know if the schmoo in question
has 'conflicts' with the context variables that are set?
Thank you for any thoughts,
jason pettiss
Catalis, Inc
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Christoph Reck <ap...@recks.org>.
>>Aaah, I get it. That's a serious performance hit but at least it'll get
>>me up and running so I can make milestone... a good tip. Thank you.
>>Then later I'll just go back and replace $ and # with \ or whatever the
>>escape is, so things are fast again.
>>
>>So what's the history behind the broken escape mechanism? It's not
>>really an escape mechanism if it can't escape itself, after all. I mean
>>the problem is, \$foo should always always always output $foo. Because
>>there is no way of knowing at template time what's in the context and so
>>there's no way of knowing if random backslashes are going to creep up.
The original idea, was that your character stream could contain "\$foo",
and therefore, if $foo was not in the context, it would emit "\$foo" on
the output.
This caused the seemingly broken escaping (which acts differently if a
reference being or not being in the context).
>
> [snip]
>
> Yeah, I have debated in a long thread about that earlier. I dare to
> state that it is, as a matter if fact... amm... so, let me call it what
> it is: plain stupid design. And most probably it will not be fixed,
> since that would be not backward compatible, of course. So don't waste
> your time thinking about it, just live with ${D} and ${H}. Who cares
> after all... this minor speed loss barely does mater.
>
Dito, just live with ${D} and ${H} and you are on the safe side.
:) Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Jason Pettiss <ja...@CatalisHealth.com>.
Well, it definitely seems better to handle the whitespace issue with
post-processing than with the grammar-- it looks like most of the "hmm
will this work" comments in the grammar all relate to munching
whitespace as part of parsing. And of course the way it's handled
currently means that if you wanted flexibility about how to handle
whitespace, you're *totally* dead in the water.
I buy the "parsing schmoo gives you schmoo" argument only so much.
Because that applies only with an empty context! You either have to
know everything that can possibly be set in the context through all time
and space, or you have to have the happy situation that a dollar sign
followed by text isn't all that common. Otherwise, you have to always
escape dollar signs. Using the ${D} hack instead of the so-called
"escape" mechanism, since the escape doesn't actually escape /parsing/. ;-)
Actually, removing the backslash mechanism from the language entirely
would probably help everyone involved-- it would eliminate sooooo many
special cases and "I'm not sure if this works" hacks, on both sides of
the fence-- Velocity template authors and Velocity developers. Man, you
want funny, look at the handling for $\!foo. Ouch, my head hurts.
I'm not really familiar with JavaCC but at this point it doesn't look
like SimpleNode is set up to allow modifications, nor does it have a
copy constructor or a clone method. So modifying the AST after parsing
is currently a no-go. For my problem, adding a "is reference a
constant" callout to the grammar file would be easy and I'm thinkin'
${D} and ${H} are gonna be hardcoded as special literals. Just because
I can. Muhahaha.
But to handle whitespace like you want-- configurably-- you'd want to
tweak an AST that's already been built, and have the grammar be
completely respectful of whitespace and create whitespace literal tokens
for you.
Definitely interesting to think about, tho... ;-)
--jason
Christoph Reck wrote:
> Jason Pettiss wrote:
>
>> Actually, after reading through the parser grammar it's pretty clear
>> this is haunting the authors. They're trying to describe an escape
>> sequence in the grammar itself which causes havoc for the poor
>> parser. Escape sequences should be handled by the scanner. Really
>> when \$ appears in the stream it should be emitted as $ in literal
>> output, with no possibility to be tokenized. Then the grammer
>> doesn't care and is smaller, the parser is faster, and you have an
>> outlet for backwards-compatible growth if in the future you need new
>> language constructs. (Imagine if #hello is in my template. And in
>> the future hello becomes a directive. It breaks me. But if I do
>> \#hello now, then I'm always covered.)
>>
>> Your point is a good one-- does the speed really matter? In most
>> cases no, it really doesn't. In my case it actually might, I don't
>> know... there are a TON of dollar signs over megabytes of content.
>> I'm just imagining an interrupted stream and a hashmap lookup for
>> each and every one... there's no notion of constant or final template
>> variables is there?
>>
>> Hmm. Anyway, thanks for the thoughts.
>
> [snip]
>
> I can see that you understood the issue with escaping and the grammar
> (not my dog-food :), the issue that parsing any schmoo should produce
> the same schmoo prohibits gobbling the backslash in a \$foo sequence.
>
> OTOH, your idea of a "notion of constant or final template variables"
> is a good one. The short answer is no :(
>
> So this could be a feature request. Could it be done during parse time
> creating the text nodes?
>
> A workaround is to walk yourself the AST from an already parsed template
> and work in the appropriate text nodes into the exising nodes before or
> after!. Is this feasible? One could fit such a thing in by wrapping
> an existing resource loader. (Side note: the post-processing of the
> AST is an idea I have been thinking about for the whitespace gobbling
> handling issue... more on this issue is in the wiki)
>
> Cheers,
> Christoph
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Christoph Reck <ap...@recks.org>.
Jason Pettiss wrote:
> Actually, after reading through the parser grammar it's pretty clear
> this is haunting the authors. They're trying to describe an escape
> sequence in the grammar itself which causes havoc for the poor parser.
> Escape sequences should be handled by the scanner. Really when \$
> appears in the stream it should be emitted as $ in literal output, with
> no possibility to be tokenized. Then the grammer doesn't care and is
> smaller, the parser is faster, and you have an outlet for
> backwards-compatible growth if in the future you need new language
> constructs. (Imagine if #hello is in my template. And in the future
> hello becomes a directive. It breaks me. But if I do \#hello now, then
> I'm always covered.)
>
> Your point is a good one-- does the speed really matter? In most cases
> no, it really doesn't. In my case it actually might, I don't know...
> there are a TON of dollar signs over megabytes of content. I'm just
> imagining an interrupted stream and a hashmap lookup for each and every
> one... there's no notion of constant or final template variables is there?
>
> Hmm. Anyway, thanks for the thoughts.
[snip]
I can see that you understood the issue with escaping and the grammar
(not my dog-food :), the issue that parsing any schmoo should produce
the same schmoo prohibits gobbling the backslash in a \$foo sequence.
OTOH, your idea of a "notion of constant or final template variables"
is a good one. The short answer is no :(
So this could be a feature request. Could it be done during parse time
creating the text nodes?
A workaround is to walk yourself the AST from an already parsed template
and work in the appropriate text nodes into the exising nodes before or
after!. Is this feasible? One could fit such a thing in by wrapping
an existing resource loader. (Side note: the post-processing of the
AST is an idea I have been thinking about for the whitespace gobbling
handling issue... more on this issue is in the wiki)
Cheers,
Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Jason Pettiss <ja...@CatalisHealth.com>.
Actually, after reading through the parser grammar it's pretty clear
this is haunting the authors. They're trying to describe an escape
sequence in the grammar itself which causes havoc for the poor parser.
Escape sequences should be handled by the scanner. Really when \$
appears in the stream it should be emitted as $ in literal output, with
no possibility to be tokenized. Then the grammer doesn't care and is
smaller, the parser is faster, and you have an outlet for
backwards-compatible growth if in the future you need new language
constructs. (Imagine if #hello is in my template. And in the future
hello becomes a directive. It breaks me. But if I do \#hello now, then
I'm always covered.)
Your point is a good one-- does the speed really matter? In most cases
no, it really doesn't. In my case it actually might, I don't know...
there are a TON of dollar signs over megabytes of content. I'm just
imagining an interrupted stream and a hashmap lookup for each and every
one... there's no notion of constant or final template variables is there?
Hmm. Anyway, thanks for the thoughts.
--jason
Daniel Dekany wrote:
>Wednesday, July 6, 2005, 5:53:54 PM, Jason Pettiss wrote:
>
>
>
>>Aaah, I get it. That's a serious performance hit but at least it'll get
>>me up and running so I can make milestone... a good tip. Thank you.
>>Then later I'll just go back and replace $ and # with \ or whatever the
>>escape is, so things are fast again.
>>
>>So what's the history behind the broken escape mechanism? It's not
>>really an escape mechanism if it can't escape itself, after all. I mean
>>the problem is, \$foo should always always always output $foo. Because
>>there is no way of knowing at template time what's in the context and so
>>there's no way of knowing if random backslashes are going to creep up.
>>
>>
>[snip]
>
>Yeah, I have debated in a long thread about that earlier. I dare to
>state that it is, as a matter if fact... amm... so, let me call it what
>it is: plain stupid design. And most probably it will not be fixed,
>since that would be not backward compatible, of course. So don't waste
>your time thinking about it, just live with ${D} and ${H}. Who cares
>after all... this minor speed loss barely does mater.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Daniel Dekany <dd...@freemail.hu>.
Wednesday, July 6, 2005, 5:53:54 PM, Jason Pettiss wrote:
> Aaah, I get it. That's a serious performance hit but at least it'll get
> me up and running so I can make milestone... a good tip. Thank you.
> Then later I'll just go back and replace $ and # with \ or whatever the
> escape is, so things are fast again.
>
> So what's the history behind the broken escape mechanism? It's not
> really an escape mechanism if it can't escape itself, after all. I mean
> the problem is, \$foo should always always always output $foo. Because
> there is no way of knowing at template time what's in the context and so
> there's no way of knowing if random backslashes are going to creep up.
[snip]
Yeah, I have debated in a long thread about that earlier. I dare to
state that it is, as a matter if fact... amm... so, let me call it what
it is: plain stupid design. And most probably it will not be fixed,
since that would be not backward compatible, of course. So don't waste
your time thinking about it, just live with ${D} and ${H}. Who cares
after all... this minor speed loss barely does mater.
--
Best regards,
Daniel Dekany
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Jason Pettiss <ja...@CatalisHealth.com>.
Aaah, I get it. That's a serious performance hit but at least it'll get
me up and running so I can make milestone... a good tip. Thank you.
Then later I'll just go back and replace $ and # with \ or whatever the
escape is, so things are fast again.
So what's the history behind the broken escape mechanism? It's not
really an escape mechanism if it can't escape itself, after all. I mean
the problem is, \$foo should always always always output $foo. Because
there is no way of knowing at template time what's in the context and so
there's no way of knowing if random backslashes are going to creep up.
Yeah, I do have a problem with this-- the template "language" I'm
converting from is actually a mish-mash of various post-processing
operations and SSIs, it's a real mess. And unfortunately for me, $ to
end of line marks a 'literal zone'-- where the actual content resides
So pretty much any word that can follow a dollar sign, does! Backslash
is, as usual, used for character escape sequences, and \$ isn't
anything, so luckily it gets ignored. But having random \'s in the code
blocks understandably throws a wrench into things...
So for now ${D} will escape $, and if ${D} ever does appear it will be
${D}{D} which will still work. Good tip, good tip. Thanks,
--jason
Christoph Reck wrote:
> Hi Jason,
>
> Velocity was originaly designed such that if you parse a template
> with 'schmoo' will output exactly a copy of the input. Any $reference
> encountered, which is not in the context should be textually copied
> to the output. So I don't understand your problem... Is there one
> when you tried parsing your input without escaping?
>
> OTOH, escaping with a backslash \$ only works partially in velocity.
> You may call it a "broken" feature.
>
> Use velocity's form of poor man's escaping, which will always work:
>
> #set ($D = '$' )
> #set( $H = '#' )
> ${D}sys.thing($H$H) just a test
>
> will then properly output:
>
> $sys.thing(##) just a test
>
>
> :) Christoph Reck
>
>
> Jason Pettiss wrote:
>
>> I'm going insane here, maybe someone familiar with the parsing of
>> velocity templates can help out. I have a document which is all
>> 'schmoo'-- it's not a velocity template... yet. I need to make sure
>> this document, when parsed by velocity, comes out exactly like it
>> is-- so that all schmoo is schmoo. But I'm having a hell of a time
>> doing this.
>>
>> I figured that it would be a simple matter of escaping every
>> backslash in the document with another backslash, and then escaping
>> every $ and # with a backslash. But this doubles the backslashes and
>> does random stuff with the dollar signs (the directives, at least,
>> all have their wings clipped).
>>
>> In particular I notice that \$foo is not guaranteed to come out as
>> $foo like I would expect. After all what good is an escape sequence
>> if it doesn't actually escape out of the parsing behavior! But I
>> find that depending on whether 'foo' happens to be set or not, \$foo
>> will either come out $foo or \$foo. This extra backslash is totally
>> uncalled for-- I have no way to escape it, and I have no way of
>> knowing whether it will appear or not since I don't actually care to
>> know what's in the document I'm handling. I mean, maybe it was
>> legitimately talking about the cost of foo, or something.
>>
>> For those curious, I'm actually converting a gigantic content
>> repository from an ancient templating language into Velocity. It
>> involves taking a document, escaping everything that Velocity might
>> think is Velocity because it's just schmoo, and then sprinkling
>> Velocity directives and variables all over the document (this IS
>> Velocity). How do I tell Velocity to piss off without having
>> backslashes randomly appear all over the place, and without actually
>> having to know if the schmoo in question has 'conflicts' with the
>> context variables that are set?
>>
>> Thank you for any thoughts,
>>
>> jason pettiss
>> Catalis, Inc
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
Re: Escapage of escapage
Posted by Christoph Reck <ap...@recks.org>.
Hi Jason,
Velocity was originaly designed such that if you parse a template
with 'schmoo' will output exactly a copy of the input. Any $reference
encountered, which is not in the context should be textually copied
to the output. So I don't understand your problem... Is there one
when you tried parsing your input without escaping?
OTOH, escaping with a backslash \$ only works partially in velocity.
You may call it a "broken" feature.
Use velocity's form of poor man's escaping, which will always work:
#set ($D = '$' )
#set( $H = '#' )
${D}sys.thing($H$H) just a test
will then properly output:
$sys.thing(##) just a test
:) Christoph Reck
Jason Pettiss wrote:
> I'm going insane here, maybe someone familiar with the parsing of
> velocity templates can help out. I have a document which is all
> 'schmoo'-- it's not a velocity template... yet. I need to make sure
> this document, when parsed by velocity, comes out exactly like it is--
> so that all schmoo is schmoo. But I'm having a hell of a time doing this.
>
> I figured that it would be a simple matter of escaping every backslash
> in the document with another backslash, and then escaping every $ and #
> with a backslash. But this doubles the backslashes and does random
> stuff with the dollar signs (the directives, at least, all have their
> wings clipped).
>
> In particular I notice that \$foo is not guaranteed to come out as $foo
> like I would expect. After all what good is an escape sequence if it
> doesn't actually escape out of the parsing behavior! But I find that
> depending on whether 'foo' happens to be set or not, \$foo will either
> come out $foo or \$foo. This extra backslash is totally uncalled for--
> I have no way to escape it, and I have no way of knowing whether it will
> appear or not since I don't actually care to know what's in the document
> I'm handling. I mean, maybe it was legitimately talking about the cost
> of foo, or something.
>
> For those curious, I'm actually converting a gigantic content repository
> from an ancient templating language into Velocity. It involves taking a
> document, escaping everything that Velocity might think is Velocity
> because it's just schmoo, and then sprinkling Velocity directives and
> variables all over the document (this IS Velocity). How do I tell
> Velocity to piss off without having backslashes randomly appear all over
> the place, and without actually having to know if the schmoo in question
> has 'conflicts' with the context variables that are set?
>
> Thank you for any thoughts,
>
> jason pettiss
> Catalis, Inc
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-dev-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-dev-help@jakarta.apache.org