You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Thejas M Nair (JIRA)" <ji...@apache.org> on 2013/08/09 04:59:48 UTC
[jira] [Commented] (HIVE-4986) Problem in creating configration in
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils
[ https://issues.apache.org/jira/browse/HIVE-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734353#comment-13734353 ]
Thejas M Nair commented on HIVE-4986:
-------------------------------------
[~subrotosanyal] This does look like a bug. Is setting core-site.xml/hdfs-site.xml in path a suitable work around for this issue ?
> Problem in creating configration in org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils
> -------------------------------------------------------------------------------------
>
> Key: HIVE-4986
> URL: https://issues.apache.org/jira/browse/HIVE-4986
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Subroto Sanyal
>
> While using AvroSerDe but, I am facing the problem:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS: hdfs://xxx:8020/abc/bbc/test.avsc, expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
> at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
> at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromHDFS(AvroSerdeUtils.java:105)
> at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:70)
> at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
> at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
> at test.subroto.HiveUtil.getDeserializer(HiveUtil.java:102)
> {noformat}
> While debugging the problem I saw a piece of code in:
> {code:title=org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.java|borderStyle=solid}
> public static Schema More ...determineSchemaOrThrowException(Properties properties)
> throws IOException, AvroSerdeException {
> String schemaString = properties.getProperty(SCHEMA_LITERAL);
> if(schemaString != null && !schemaString.equals(SCHEMA_NONE))
> return Schema.parse(schemaString);
>
> // Try pulling directly from URL
> schemaString = properties.getProperty(SCHEMA_URL);
> if(schemaString == null || schemaString.equals(SCHEMA_NONE))
> throw new AvroSerdeException(EXCEPTION_MESSAGE);
>
> try {
> if(schemaString.toLowerCase().startsWith("hdfs://"))
> return getSchemaFromHDFS(schemaString, new Configuration());
> } catch(IOException ioe) {
> throw new AvroSerdeException("Unable to read schema from HDFS: " + schemaString, ioe);
> }
>
> return Schema.parse(new URL(schemaString).openStream());
> }
> {code}
> Here the configuration passed from the Deserilaizer is completely ignored and a new Configuration object is created.
> In my opinion the AvroSerDe should honor the Configuration being passed to Deserializer interface ; rather than creating a new one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira