You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2010/03/31 01:50:27 UTC
[jira] Created: (PIG-1341) Cannot convert DataByeArray to Chararray
and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED 20
Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED 20
-------------------------------------------------------------------------------------------------
Key: PIG-1341
URL: https://issues.apache.org/jira/browse/PIG-1341
Project: Pig
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Viraj Bhat
Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
{code}
raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
--filter out null columns
A = filter raw by col1#'bcookie' is not null;
B = foreach A generate col1#'bcookie' as reqcolumn;
describe B;
--B: {regcolumn: bytearray}
X = limit B 5;
dump X;
B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
describe B;
--B: {convertedcol: chararray}
X = limit B 5;
dump X;
{code}
The first dump produces:
(36co9b55onr8s)
(36co9b55onr8s)
(36hilul5oo1q1)
(36hilul5oo1q1)
(36l4cj15ooa8a)
The second dump produces:
()
()
()
()
()
It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1341:
------------------------------
Fix Version/s: (was: 0.7.0)
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) Cannot convert DataByeArray to Chararray
and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-1341:
----------------------------
Component/s: impl
Summary: Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED (was: Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED 20)
> Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928887#action_12928887 ]
Alan Gates commented on PIG-1341:
---------------------------------
We can mark BinStorage as deprecated if we want, though I don't think we should without a decent sequence file load/store function. But we can't remove it at the moment, as people use it.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Fix For: 0.9.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) Cannot convert DataByeArray to Chararray
and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1341:
--------------------------------
Fix Version/s: 0.7.0
> Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Fix For: 0.7.0
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852565#action_12852565 ]
Daniel Dai commented on PIG-1341:
---------------------------------
I am thinking about encoding lineage info into the BinStorage file header, so even after we "dump and load", we still maintain the lineage information.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1341:
------------------------------
Attachment: PIG-1341.patch
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1341:
------------------------------
Status: Patch Available (was: Open)
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1341) Cannot convert DataByeArray to
Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-1341:
-----------------------------------
Assignee: Richard Ding
> Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852515#action_12852515 ]
Daniel Dai commented on PIG-1341:
---------------------------------
Have a discussion with Alan and Richard, we felt that caster for BinStorage does not make sense. We don't know how to cast bytearray datatype for BinStorage. In the intermediate storage case, we will find the original loader, and use lineage for that loader to convert bytearray. But if user use the BinStorage directly, we have no idea what bytearray means. So the suggestion is we don't give caster to BinStorage. The implication is that if user want to use BinStorage as a temporary store, in some cases, it will fail.
Here is a sample script which will be broken if we make this change:
script 1:
{code}
a = load '1.txt';
b = order a by $0;
store b into 'temp.out' using BinStorage(); -- store in BinStorage format with the datatype bytearray
{code}
script 2:
{code}
a = load 'temp.out' using BinStorage();
b = foreach a generate $0+$1; -- here we will need a caster, but BinStorage does not have it, we will fail
{code}
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859453#action_12859453 ]
Alan Gates commented on PIG-1341:
---------------------------------
I agree with Ashutosh. We do not want BinStorage tracking data lineage. In the case where Pig is using BinStorage (or whatever) for moving data between MR jobs then Pig can figure out the correct cast function to use and apply it. For cases such as the one here where users are storing data using BinStorage and then in a separate Pig Latin script reading it (and thus loosing the type information) it is the users responsibility to correctly cast the data before storing it in BinStorage. As a general case I do not think we can expect load and store functions to track data lineage across Pig Latin scripts.
I propose we close this as won't fix.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928884#action_12928884 ]
Viraj Bhat commented on PIG-1341:
---------------------------------
Hi Alan,
I think we should stop exposing BinStorage to users otherwise this will keep happening.
We should document this behavior of BinStorage if we are allowing users to use this load-store class.
Viraj
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Fix For: 0.9.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859157#action_12859157 ]
Ashutosh Chauhan commented on PIG-1341:
---------------------------------------
I think BinStorage is an internal way of moving data around in Pig and it should be treated that way. I think we should discourage its usage to user. Otherwise, we need to add capabilities as the one requested here. Important impact of making such a change is that we can't then swap out BinStorage with other storage mechanisms. If Avro (or protobuf or whatever) proved to be a better replacement for BinStorage, then we cant just swap them in place of BinStorage, unless we add to them all the capabilities that BinStorage has. Therefore, I suggest to keep capabilities of BinStorage to minimal.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852308#action_12852308 ]
Hadoop QA commented on PIG-1341:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12440415/PIG-1341.patch
against trunk revision 929737.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/266/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/266/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/266/console
This message is automatically generated.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1341:
--------------------------------
Fix Version/s: 0.9.0
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.9.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reassigned PIG-1341:
-------------------------------
Assignee: Alan Gates (was: Richard Ding)
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Fix For: 0.9.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1341:
----------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Alan Gates
> Fix For: 0.9.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1341) BinStorage cannot convert DataByteArray
to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1341:
------------------------------
Summary: BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED (was: Cannot convert DataByeArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED)
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1341) BinStorage cannot convert
DataByteArray to Chararray and results in
FIELD_DISCARDED_TYPE_CONVERSION_FAILED
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852521#action_12852521 ]
Daniel Dai commented on PIG-1341:
---------------------------------
Also if we doing that, BinStorage is not "a reliable way to dump data and load it, without having to explicitly list all the fields and figure out their parts" (though it is already not). I think we shall provide some way for this use case.
> BinStorage cannot convert DataByteArray to Chararray and results in FIELD_DISCARDED_TYPE_CONVERSION_FAILED
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-1341
> URL: https://issues.apache.org/jira/browse/PIG-1341
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1341.patch
>
>
> Script reads in BinStorage data and tries to convert a column which is in DataByteArray to Chararray.
> {code}
> raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
> --filter out null columns
> A = filter raw by col1#'bcookie' is not null;
> B = foreach A generate col1#'bcookie' as reqcolumn;
> describe B;
> --B: {regcolumn: bytearray}
> X = limit B 5;
> dump X;
> B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
> describe B;
> --B: {convertedcol: chararray}
> X = limit B 5;
> dump X;
> {code}
> The first dump produces:
> (36co9b55onr8s)
> (36co9b55onr8s)
> (36hilul5oo1q1)
> (36hilul5oo1q1)
> (36l4cj15ooa8a)
> The second dump produces:
> ()
> ()
> ()
> ()
> ()
> It also throws an error message: FIELD_DISCARDED_TYPE_CONVERSION_FAILED 5 time(s).
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.