You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Johannes Schwenk <jo...@adition.com> on 2013/10/21 19:05:25 UTC
Pig Schema contains a name that is not allowed in Avro
Hi!
I'm getting the following error when running the script below in grunt
(pig 0.12.0):
DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
w:chararray, x:chararray);
STORE data into '/path/to/output' USING AvroStorageSTORE();
2013-10-21 18:50:15,491 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
Details at logfile: pig_1382374188771.log
Logfile contains:
Pig Schema contains a name that is not allowed in Avro
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias data
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
at org.apache.pig.PigServer.execute(PigServer.java:1289)
at org.apache.pig.PigServer.access$400(PigServer.java:125)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
... 13 more
Caused by: java.io.IOException: Pig Schema contains a name that is not
allowed in Avro
at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
at
org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
... 26 more
Avro schema looks like:
{
"name": "entry",
"namespace": "a.b.c",
"type": "record",
"doc": "A Log",
"fields": [
{"name": "a", "type": "long"},
{"name": "b", "type": "long"},
{"name": "c" "type": "int"},
{"name": "d" "type": {
"name": "d"
"type": "enum",
"symbols": [
"A",
"B",
"C",
"D",
"E"
],
"doc": "Some doc"
}
},
{"name": "e" "type": "int"},
{"name": "f", "type": "int"},
{"name": "g", "type": "int"},
{"name": "h", "type": ["null", "long"], "default": null},
{"name": "i", "type": ["null", "string"], "default": null},
{"name": "j", "type": "int"},
{"name": "k", "type": "int"},
{"name": "l", "type": "int"},
{"name": "m", "type": ["null", "int"], "default": null},
{"name": "n", "type": ["null", "int"], "default": null},
{"name": "o", "type": ["null", "int"], "default": null},
{"name": "p", "type": ["null", "int"], "default": null},
{"name": "q", "type": ["null", "int"], "default": null},
{"name": "r", "type": ["null", "int"], "default": null},
{"name": "s", "type": ["null", "int"], "default": null},
{"name": "t", "type": "boolean"},
{"name": "u", "type": "int"},
{"name": "v", "type": "string"},
{"name": "w", "type": ["null", "string"], "default": null},
{"name": "x", "type": "string"}
]
}
Had to redact the names because of confidentiality.
Do you have any idea what the problem could be? The error message seems
misleading...
Greetings,
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Joachim Schneidmadl
UStIDNr.: DE 218 858 434
Re: Pig Schema contains a name that is not allowed in Avro
Posted by Ruslan Al-Fakikh <me...@gmail.com>.
Hey Johannes!
Have you solved the problem? I also see it.
But I don't see it when I use the schema as a string to AvroStorage
parameter. I see it only when I try to use an external schema file. And if
I specify a non-existent external schema file, the error is the same.
Ruslan
On Tue, Oct 22, 2013 at 12:16 PM, Johannes Schwenk <
johannes.schwenk@adition.com> wrote:
> Thanks for your answer!
>
> Actually the Avro schema is valid and I can load data with it. The error
> message states, that pig has a problem with the Pig schema, which has no
> duplicate names.
>
> Johannes
>
> Am 21.10.2013 19:29, schrieb j.barrett Strausser:
> > I'd imagime it is having an issue with the duplicate 'd' names? That is
> my
> > un-informed guess though.
> >
> >
> > On Mon, Oct 21, 2013 at 1:05 PM, Johannes Schwenk <
> > johannes.schwenk@adition.com> wrote:
> >
> >> Hi!
> >>
> >> I'm getting the following error when running the script below in grunt
> >> (pig 0.12.0):
> >>
> >> DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
> >> data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
> >> e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
> >> n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
> >> w:chararray, x:chararray);
> >> STORE data into '/path/to/output' USING AvroStorageSTORE();
> >>
> >> 2013-10-21 18:50:15,491 [main] INFO
> >> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> >> script: UNKNOWN
> >> 2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> >> ERROR 2116:
> >> <line 3, column 0> Output Location Validation Failed for:
> >> 'hdfs://path/to/output More info to follow:
> >> Pig Schema contains a name that is not allowed in Avro
> >> Details at logfile: pig_1382374188771.log
> >>
> >> Logfile contains:
> >>
> >> Pig Schema contains a name that is not allowed in Avro
> >>
> >> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> >> to store alias data
> >> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
> >> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >> at
> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
> >> at
> >>
> >>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >> at org.apache.pig.Main.run(Main.java:495)
> >> at org.apache.pig.Main.main(Main.java:111)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> >> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
> >> <line 3, column 0> Output Location Validation Failed for:
> >> 'hdfs://path/to/output More info to follow:
> >> Pig Schema contains a name that is not allowed in Avro
> >> at
> >>
> >>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
> >> at
> >>
> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> >> at
> >>
> >>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
> >> at
> >>
> >>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> >> at
> >>
> >>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> >> at
> >> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
> >> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> >> at
> >>
> >>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
> >> at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> >> at
> >> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> >> at org.apache.pig.PigServer.execute(PigServer.java:1289)
> >> at org.apache.pig.PigServer.access$400(PigServer.java:125)
> >> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
> >> ... 13 more
> >> Caused by: java.io.IOException: Pig Schema contains a name that is not
> >> allowed in Avro
> >> at
> >>
> >>
> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
> >> at
> >>
> >>
> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
> >> at
> >> org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
> >> at
> >>
> >>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
> >> ... 26 more
> >>
> >>
> >> Avro schema looks like:
> >>
> >> {
> >> "name": "entry",
> >> "namespace": "a.b.c",
> >> "type": "record",
> >> "doc": "A Log",
> >> "fields": [
> >> {"name": "a", "type": "long"},
> >> {"name": "b", "type": "long"},
> >> {"name": "c" "type": "int"},
> >> {"name": "d" "type": {
> >> "name": "d"
> >> "type": "enum",
> >> "symbols": [
> >> "A",
> >> "B",
> >> "C",
> >> "D",
> >> "E"
> >> ],
> >> "doc": "Some doc"
> >> }
> >> },
> >> {"name": "e" "type": "int"},
> >> {"name": "f", "type": "int"},
> >> {"name": "g", "type": "int"},
> >> {"name": "h", "type": ["null", "long"], "default":
> null},
> >> {"name": "i", "type": ["null", "string"], "default":
> null},
> >> {"name": "j", "type": "int"},
> >> {"name": "k", "type": "int"},
> >> {"name": "l", "type": "int"},
> >> {"name": "m", "type": ["null", "int"], "default": null},
> >> {"name": "n", "type": ["null", "int"], "default": null},
> >> {"name": "o", "type": ["null", "int"], "default": null},
> >> {"name": "p", "type": ["null", "int"], "default": null},
> >> {"name": "q", "type": ["null", "int"], "default": null},
> >> {"name": "r", "type": ["null", "int"], "default": null},
> >> {"name": "s", "type": ["null", "int"], "default": null},
> >> {"name": "t", "type": "boolean"},
> >> {"name": "u", "type": "int"},
> >> {"name": "v", "type": "string"},
> >> {"name": "w", "type": ["null", "string"], "default":
> null},
> >> {"name": "x", "type": "string"}
> >> ]
> >> }
> >>
> >>
> >> Had to redact the names because of confidentiality.
> >>
> >> Do you have any idea what the problem could be? The error message seems
> >> misleading...
> >>
> >> Greetings,
> >>
> >> Johannes Schwenk
> >>
> >> --
> >> Softwareentwickler (Reporting)
> >> ________________________________________________________
> >>
> >> ADITION technologies AG
> >> Schwarzwaldstraße 78b
> >> 79117 Freiburg
> >>
> >> http://www.adition.com
> >>
> >> T +49 / (0)761 / 88147 - 30
> >> F +49 / (0)761 / 88147 - 77
> >> SUPPORT +49 / (0)1805 - ADITION
> >>
> >> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
> >>
> >> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> >> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus
> Schlüter
> >> Aufsichtsratsvorsitzender: Joachim Schneidmadl
> >> UStIDNr.: DE 218 858 434
> >>
> >>
> >
> >
>
>
>
> Johannes Schwenk
>
> --
> Softwareentwickler (Reporting)
> ________________________________________________________
>
> ADITION technologies AG
> Schwarzwaldstraße 78b
> 79117 Freiburg
>
> http://www.adition.com
>
> T +49 / (0)761 / 88147 - 30
> F +49 / (0)761 / 88147 - 77
> SUPPORT +49 / (0)1805 - ADITION
>
> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>
> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
> Aufsichtsratsvorsitzender: Joachim Schneidmadl
> UStIDNr.: DE 218 858 434
>
>
Re: Pig Schema contains a name that is not allowed in Avro
Posted by Johannes Schwenk <jo...@adition.com>.
Thanks for your answer!
Actually the Avro schema is valid and I can load data with it. The error
message states, that pig has a problem with the Pig schema, which has no
duplicate names.
Johannes
Am 21.10.2013 19:29, schrieb j.barrett Strausser:
> I'd imagime it is having an issue with the duplicate 'd' names? That is my
> un-informed guess though.
>
>
> On Mon, Oct 21, 2013 at 1:05 PM, Johannes Schwenk <
> johannes.schwenk@adition.com> wrote:
>
>> Hi!
>>
>> I'm getting the following error when running the script below in grunt
>> (pig 0.12.0):
>>
>> DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
>> data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
>> e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
>> n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
>> w:chararray, x:chararray);
>> STORE data into '/path/to/output' USING AvroStorageSTORE();
>>
>> 2013-10-21 18:50:15,491 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: UNKNOWN
>> 2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>> ERROR 2116:
>> <line 3, column 0> Output Location Validation Failed for:
>> 'hdfs://path/to/output More info to follow:
>> Pig Schema contains a name that is not allowed in Avro
>> Details at logfile: pig_1382374188771.log
>>
>> Logfile contains:
>>
>> Pig Schema contains a name that is not allowed in Avro
>>
>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>> to store alias data
>> at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>> at
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
>> at
>>
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
>> at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
>> at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>> at org.apache.pig.Main.run(Main.java:495)
>> at org.apache.pig.Main.main(Main.java:111)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
>> <line 3, column 0> Output Location Validation Failed for:
>> 'hdfs://path/to/output More info to follow:
>> Pig Schema contains a name that is not allowed in Avro
>> at
>>
>> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
>> at
>> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
>> at
>>
>> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
>> at
>>
>> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>> at
>>
>> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>> at
>> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
>> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>> at
>>
>> org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
>> at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
>> at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
>> at
>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
>> at org.apache.pig.PigServer.execute(PigServer.java:1289)
>> at org.apache.pig.PigServer.access$400(PigServer.java:125)
>> at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
>> ... 13 more
>> Caused by: java.io.IOException: Pig Schema contains a name that is not
>> allowed in Avro
>> at
>>
>> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
>> at
>>
>> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
>> at
>> org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
>> at
>>
>> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
>> ... 26 more
>>
>>
>> Avro schema looks like:
>>
>> {
>> "name": "entry",
>> "namespace": "a.b.c",
>> "type": "record",
>> "doc": "A Log",
>> "fields": [
>> {"name": "a", "type": "long"},
>> {"name": "b", "type": "long"},
>> {"name": "c" "type": "int"},
>> {"name": "d" "type": {
>> "name": "d"
>> "type": "enum",
>> "symbols": [
>> "A",
>> "B",
>> "C",
>> "D",
>> "E"
>> ],
>> "doc": "Some doc"
>> }
>> },
>> {"name": "e" "type": "int"},
>> {"name": "f", "type": "int"},
>> {"name": "g", "type": "int"},
>> {"name": "h", "type": ["null", "long"], "default": null},
>> {"name": "i", "type": ["null", "string"], "default": null},
>> {"name": "j", "type": "int"},
>> {"name": "k", "type": "int"},
>> {"name": "l", "type": "int"},
>> {"name": "m", "type": ["null", "int"], "default": null},
>> {"name": "n", "type": ["null", "int"], "default": null},
>> {"name": "o", "type": ["null", "int"], "default": null},
>> {"name": "p", "type": ["null", "int"], "default": null},
>> {"name": "q", "type": ["null", "int"], "default": null},
>> {"name": "r", "type": ["null", "int"], "default": null},
>> {"name": "s", "type": ["null", "int"], "default": null},
>> {"name": "t", "type": "boolean"},
>> {"name": "u", "type": "int"},
>> {"name": "v", "type": "string"},
>> {"name": "w", "type": ["null", "string"], "default": null},
>> {"name": "x", "type": "string"}
>> ]
>> }
>>
>>
>> Had to redact the names because of confidentiality.
>>
>> Do you have any idea what the problem could be? The error message seems
>> misleading...
>>
>> Greetings,
>>
>> Johannes Schwenk
>>
>> --
>> Softwareentwickler (Reporting)
>> ________________________________________________________
>>
>> ADITION technologies AG
>> Schwarzwaldstraße 78b
>> 79117 Freiburg
>>
>> http://www.adition.com
>>
>> T +49 / (0)761 / 88147 - 30
>> F +49 / (0)761 / 88147 - 77
>> SUPPORT +49 / (0)1805 - ADITION
>>
>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>
>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
>> Aufsichtsratsvorsitzender: Joachim Schneidmadl
>> UStIDNr.: DE 218 858 434
>>
>>
>
>
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Joachim Schneidmadl
UStIDNr.: DE 218 858 434
Re: Pig Schema contains a name that is not allowed in Avro
Posted by "j.barrett Strausser" <j....@gmail.com>.
I'd imagime it is having an issue with the duplicate 'd' names? That is my
un-informed guess though.
On Mon, Oct 21, 2013 at 1:05 PM, Johannes Schwenk <
johannes.schwenk@adition.com> wrote:
> Hi!
>
> I'm getting the following error when running the script below in grunt
> (pig 0.12.0):
>
> DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
> data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
> e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
> n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
> w:chararray, x:chararray);
> STORE data into '/path/to/output' USING AvroStorageSTORE();
>
> 2013-10-21 18:50:15,491 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2116:
> <line 3, column 0> Output Location Validation Failed for:
> 'hdfs://path/to/output More info to follow:
> Pig Schema contains a name that is not allowed in Avro
> Details at logfile: pig_1382374188771.log
>
> Logfile contains:
>
> Pig Schema contains a name that is not allowed in Avro
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> to store alias data
> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
> at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:495)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
> <line 3, column 0> Output Location Validation Failed for:
> 'hdfs://path/to/output More info to follow:
> Pig Schema contains a name that is not allowed in Avro
> at
>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
> at
> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at
>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
> at
>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> at
>
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> at
> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at
>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
> at
>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> at org.apache.pig.PigServer.execute(PigServer.java:1289)
> at org.apache.pig.PigServer.access$400(PigServer.java:125)
> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
> ... 13 more
> Caused by: java.io.IOException: Pig Schema contains a name that is not
> allowed in Avro
> at
>
> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
> at
>
> org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
> at
> org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
> at
>
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
> ... 26 more
>
>
> Avro schema looks like:
>
> {
> "name": "entry",
> "namespace": "a.b.c",
> "type": "record",
> "doc": "A Log",
> "fields": [
> {"name": "a", "type": "long"},
> {"name": "b", "type": "long"},
> {"name": "c" "type": "int"},
> {"name": "d" "type": {
> "name": "d"
> "type": "enum",
> "symbols": [
> "A",
> "B",
> "C",
> "D",
> "E"
> ],
> "doc": "Some doc"
> }
> },
> {"name": "e" "type": "int"},
> {"name": "f", "type": "int"},
> {"name": "g", "type": "int"},
> {"name": "h", "type": ["null", "long"], "default": null},
> {"name": "i", "type": ["null", "string"], "default": null},
> {"name": "j", "type": "int"},
> {"name": "k", "type": "int"},
> {"name": "l", "type": "int"},
> {"name": "m", "type": ["null", "int"], "default": null},
> {"name": "n", "type": ["null", "int"], "default": null},
> {"name": "o", "type": ["null", "int"], "default": null},
> {"name": "p", "type": ["null", "int"], "default": null},
> {"name": "q", "type": ["null", "int"], "default": null},
> {"name": "r", "type": ["null", "int"], "default": null},
> {"name": "s", "type": ["null", "int"], "default": null},
> {"name": "t", "type": "boolean"},
> {"name": "u", "type": "int"},
> {"name": "v", "type": "string"},
> {"name": "w", "type": ["null", "string"], "default": null},
> {"name": "x", "type": "string"}
> ]
> }
>
>
> Had to redact the names because of confidentiality.
>
> Do you have any idea what the problem could be? The error message seems
> misleading...
>
> Greetings,
>
> Johannes Schwenk
>
> --
> Softwareentwickler (Reporting)
> ________________________________________________________
>
> ADITION technologies AG
> Schwarzwaldstraße 78b
> 79117 Freiburg
>
> http://www.adition.com
>
> T +49 / (0)761 / 88147 - 30
> F +49 / (0)761 / 88147 - 77
> SUPPORT +49 / (0)1805 - ADITION
>
> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>
> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
> Aufsichtsratsvorsitzender: Joachim Schneidmadl
> UStIDNr.: DE 218 858 434
>
>
--
https://github.com/bearrito
@deepbearrito