Parameter Substitution In Call to STORE

[Andreas Paepcke <pa...@cs.stanford.edu>]

Hi,

I'm trying to pass a STORE target directory and file
into a script, providing a default. My problem
is with the parameter substitution in the STORE
function. Below are three versions of the STORE
command. Each generates a different fault.

What's the right way to do this?

Thanks!

Andreas


%default TARGET_DIR '/foo/bar';
%default URL_INDEX_FILE '${CRAWL_SOURCE}URL';

URLMap = ...;

-- The following creates directory ${TARGET_DIR} in HDFS, which is very
difficult
-- to remove, and is not what I want:
STORE URLMap INTO '${TARGET_DIR}/${URL_INDEX_FILE}'
             USING PigStorage;

-- This version unsurprisingly generates error: mismatched input '$'
expecting QUOTEDSTRING
STORE URLMap INTO ${TARGET_DIR}/${URL_INDEX_FILE}
             USING PigStorage

-- This one generates an Illegal group reference error (trace below).
-- Some Web research suggests this error might occur during the use
-- of the Java matcher facility:
STORE URLMap INTO '$TARGET_DIR/$URL_INDEX_FILE'
             USING PigStorage;

Stack trace of last version:

2011-04-08 18:58:24,517 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Illegal group reference
2011-04-08 18:58:24,517 [main] ERROR org.apache.pig.Main -
java.lang.IllegalArgumentException: Illegal group reference
        at java.util.regex.Matcher.appendReplacement(Matcher.java:713)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:861)
        at java.lang.String.replaceFirst(String.java:2146)
        at
org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:236)
        at
org.apache.pig.tools.parameters.PigFileParser.input(PigFileParser.java:65)
        at
org.apache.pig.tools.parameters.PigFileParser.Parse(PigFileParser.java:43)
        at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:105)
        at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:98)
        at org.apache.pig.Main.runParamPreprocessor(Main.java:692)
        at org.apache.pig.Main.run(Main.java:519)
        at org.apache.pig.Main.main(Main.java:108)