You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2018/03/29 05:48:00 UTC
[jira] [Commented] (CRUNCH-668) From.avroFile do not support
globbing patterns (GenericData based overloads)
[ https://issues.apache.org/jira/browse/CRUNCH-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418475#comment-16418475 ]
Josh Wills commented on CRUNCH-668:
-----------------------------------
Hey [~clement@unportant.info]! Thanks for this; I applied it to my master branch and got a few failures in tests in crunch-core and was wondering if you saw the same thing:
Tests in error:
AvroParquetFileSourceTargetIT.testCustomReadSchemaGeneric_FieldSuperset:258 » Runtime
AvroParquetFileSourceTargetIT.testCustomReadSchemaWithProjection:297 » Runtime
AvroParquetFileSourceTargetIT.testCustomReadSchema_FieldSubset:221 » Runtime E...
> From.avroFile do not support globbing patterns (GenericData based overloads)
> ----------------------------------------------------------------------------
>
> Key: CRUNCH-668
> URL: https://issues.apache.org/jira/browse/CRUNCH-668
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.15.0
> Reporter: Clément MATHIEU
> Assignee: Josh Wills
> Priority: Major
> Attachments: 0001-CRUNCH-668-Support-globbing-patterns-in-From-avroFil.patch
>
>
> GenericData based overloads of {{From.avroFile}} throws a RuntimeException when a globbing pattern is provided. I see no reason to not support globbing patterns here as it works fine with {{textFile}} and SpecificData based overloads.
> The issue is that the code extracting Avro schema from the first file use {{listStatus}} rather than {{globStatus}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)