You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2011/07/29 00:28:10 UTC
[jira] [Created] (PIG-2195) AvroStorage fails to STORE when LOADing
via PigStorage
AvroStorage fails to STORE when LOADing via PigStorage
------------------------------------------------------
Key: PIG-2195
URL: https://issues.apache.org/jira/browse/PIG-2195
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
{{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
The Pig script in this section of the documentation shows an example like this that fails:
http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
A workaround currently exists to produce avro from TSVs like this:
{noformat}
avro = LOAD 'inputPath/' AS (foo);
STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
'{"data":"data_file.avro",
"same":"data_file.avro", "field0":"def:bar"}');
{noformat}
This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
# Read schema from an existing data file:
{noformat}
'{"data":"data_file.avro", "field0":"def:bar"}');
{noformat}
# Read schema from an existing schema file:
{noformat}
'{"schema_file":"data_file.avsc", "field0":"def:bar"}');
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2195) AvroStorage fails to STORE when
LOADing via PigStorage
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084343#comment-13084343 ]
Alan Gates commented on PIG-2195:
---------------------------------
I'll be reviewing this patch.
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2195) AvroStorage fails to STORE when
LOADing via PigStorage
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078394#comment-13078394 ]
Bill Graham commented on PIG-2195:
----------------------------------
After spending many hours getting my head around what AvroStorage does, I've learned a few things some of which lead me to amend my initial assessment of the issue:
# Reading from PigStorage and writing via AvroStorage works as advertised when using Avro 1.4.1. The ClassCastException I initially referenced happens when using Avro 1.5.1. Also the current trunk doesn't build against 1.5.1. We can address this in a separate JIRA.
# There is a nasty hidden bug that causes the unit tests to not run in isolation. This can cause newly added unit tests or re-ordered tests to fail. The issue is that {{PigSchema2Avro}} has a {{static int tupleIndex}} that it increments after each call to {{getRecordName()}} and the tests expect record names to have a certain {{tupleIndex}} value included. As a temporary hack I've added a {{public static void setTupleIndex(int index)}} method to that class to allow unit tests to reset it, but this static int approach should really be revisited.
# I've added additional unit tests for reading from a text file and producing Avro.
# I've added support for passing a {{schema_file}} value instead of a data file as shown above.
I'll upload a patch shortly.
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2195) AvroStorage fails to STORE when LOADing
via PigStorage
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated PIG-2195:
-----------------------------
Attachment: expected_testRecordSplitFromText2.avro
expected_testRecordSplitFromText1.avro
PIG-2195_1.patch
Attached is a first patch that has:
* More unit tests reading from text files.
* A fix to how unit tests are run as described above.
* Support for specifying a JSON {{schema_file}}.
Also included are 2 new expected test result files that are needed for one of the new tests. The should live here:
{{contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files}}
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2195) AvroStorage fails to STORE when
LOADing via PigStorage
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080300#comment-13080300 ]
Bill Graham commented on PIG-2195:
----------------------------------
This first patch is ready for review FYI. Also FYI, I've created PIG-2202 to track upgrading AvroStorage to work with Avro 1.5.1 and PIG-2201 to track a separate issue with the unit tests.
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2195) AvroStorage fails to STORE when LOADing
via PigStorage
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated PIG-2195:
-----------------------------
Release Note: AvroStorage support for using a schema from an Avro schema file.
Status: Patch Available (was: Open)
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2195) AvroStorage fails to STORE when LOADing
via PigStorage
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-2195:
----------------------------
Resolution: Fixed
Fix Version/s: 0.10
Status: Resolved (was: Patch Available)
Patch checked in. Thanks Bill.
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Fix For: 0.10
>
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2195) AvroStorage fails to STORE when
LOADing via PigStorage
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084462#comment-13084462 ]
Bill Graham commented on PIG-2195:
----------------------------------
Thanks Alan. Regarding the naming of the new {{schema_file}} field, with the recent commit of AVRO-872, Avro will supports parsing multiple schema files in 1.6.0. I'd like to contribute a patch to {{AvroStorage}} to support that at some point.
With that in mind, should {{schema_file}} instead be {{schema_files}} so it can at some point accept a comma-separated list perhaps? Or we could allow multiple files to be passed to {{schema_file}} at some point. Or we can later introduce {{schema_files}} with an {{s}}. Many options, but something to plan for now.
> AvroStorage fails to STORE when LOADing via PigStorage
> ------------------------------------------------------
>
> Key: PIG-2195
> URL: https://issues.apache.org/jira/browse/PIG-2195
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
> Attachments: PIG-2195_1.patch, expected_testRecordSplitFromText1.avro, expected_testRecordSplitFromText2.avro
>
>
> Reading data via {{PigStorage}} and writing it via {{AvroStorage}} fails with an exception like this
> {{java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord}}
> The Pig script in this section of the documentation shows an example like this that fails:
> http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.
> A workaround currently exists to produce avro from TSVs like this:
> {noformat}
> avro = LOAD 'inputPath/' AS (foo);
> STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
> '{"data":"data_file.avro",
> "same":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> This is redundant though and {{data}} and {{same}} seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.
> # Read schema from an existing data file:
> {noformat}
> '{"data":"data_file.avro", "field0":"def:bar"}');
> {noformat}
> # Read schema from an existing schema file:
> {noformat}
> '{"schema_file":"data_file.avsc", "field0":"def:bar"}');
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira