You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Ryan Blue (JIRA)" <ji...@apache.org> on 2015/12/10 18:36:11 UTC

[jira] [Commented] (PARQUET-402) Apache Pig cannot store Map data type into Parquet format

    [ https://issues.apache.org/jira/browse/PARQUET-402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051312#comment-15051312 ] 

Ryan Blue commented on PARQUET-402:
-----------------------------------

What was the type of my_map before you used the explicit type? It seems to me that the type should be something we can work with. I wonder if there is still a bug here we should address.

Also, could you open an issue for the second problem you found, that you now have to use the full path to ParquetStorer? That's probably something we can fix easily and (I'm guessing) happened when we moved to org.apache packages.

> Apache Pig cannot store Map data type into Parquet format
> ---------------------------------------------------------
>
>                 Key: PARQUET-402
>                 URL: https://issues.apache.org/jira/browse/PARQUET-402
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-pig
>    Affects Versions: 1.6.0, 1.8.1
>            Reporter: Jerry Ylilammi
>
> Trying to store simple map with two entries gives me following exception:
> {code}table_with_map_data: {my_map: map[]}
> 2015-12-10 11:58:54,478 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2015-12-10 11:58:54,498 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Invalid map Schema, schema should contain exactly one field: my_map: map{code}
> For example taking any input and doing this gives me the exception:
> {code}table_with_map_data = FOREACH random_data GENERATE TOMAP('123', 'hello', '456', 'world') as (my_map);
> DESCRIBE table_with_map_data;
> STORE table_with_map_data INTO '...' USING ParquetStorer();{code}
> I'm using latest version of Pig: Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
> and Parquet: parquet-pig-bundle-1.6.0.jar
> EDIT: I noticed Parquet 1.8.1 is out. I switched to it and were forced to update the pig script to use full path with ParquetStorer. However this gives me same error as 1.6.0.
> {code}STORE table_with_map_data INTO '/Users/jerry/tmp/parquet/output/parquet' USING org.apache.parquet.pig.ParquetStorer();{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)