[jira] [Commented] (AVRO-1682) Support for bigdecimal type

["Ryan Blue (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590807#comment-14590807 ] 

Ryan Blue commented on AVRO-1682:
---------------------------------

Hi [~jacopo.moronato], thanks for reporting the issue. This actually isn't too surprising because we just added support for decimals in AVRO-1497. The next Avro release, 1.7.8, should include the Avro-side support and then libraries like Pig can start integrating with it. That can't happen quite yet since the release isn't out, but the Avro side is complete and just waiting for 1.7.8. I think the next step is to report this to the Pig community so they can add the integration that is needed in that project. Would you mind opening an issue for the Pig project? Then I'll link this one to it and close this since it's done! Thanks!

> Support for bigdecimal type
> ---------------------------
>
>                 Key: AVRO-1682
>                 URL: https://issues.apache.org/jira/browse/AVRO-1682
>             Project: Avro
>          Issue Type: Wish
>    Affects Versions: 1.7.6
>         Environment: Apache Pig version 0.12.0-cdh5.4.2
>            Reporter: Jacopo Moronato
>            Priority: Minor
>
> I'm using a pig script to:
> 1) load a semicolon-separate file
> 2) convert some chararray fields to bigdecimal
> 3) store into avro file.
> Point 3 fails raising this exception:
>  
>  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
>   at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
>   at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
>   at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
>   at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
>   at org.apache.pig.PigServer.compilePp(PigServer.java:1392)
>   at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1317)
>   at org.apache.pig.PigServer.execute(PigServer.java:1309)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
>   at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
>   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
>   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:478)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>   Caused by: java.io.IOException: Don't know how to encode type bigdecimal in schema
>   
>   at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceFieldSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:511)
>   at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:336)
>   at org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:486)
>   at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
>   ... 48 more
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)