You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2010/02/19 00:27:28 UTC
[jira] Created: (PIG-1243) Passing Complex map types to and from
streaming causes a problem
Passing Complex map types to and from streaming causes a problem
----------------------------------------------------------------
Key: PIG-1243
URL: https://issues.apache.org/jira/browse/PIG-1243
Project: Pig
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Viraj Bhat
Fix For: 0.7.0
I have a program which generates different types of Maps fields and stores it into PigStorage.
{code}
A = load '/user/viraj/three.txt' using PigStorage();
B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}] as d;
store B into '/user/viraj/pigtest' using PigStorage();
{code}
Now I test the previous output in the below script to make sure I have the right results. I also pass this data to a Perl script and I observe that the complex Map types I have generated, are lost when I get the result back.
{code}
DEFINE CMD `simple.pl` SHIP('simple.pl');
A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);
B = foreach A generate $0, $1, $2;
dump B;
C = foreach A generate (chararray)simpleFields#'a' as value, $0,$1,$2;
D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);
dump D;
{code}
dumping B results in:
([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
dumping D results in:
([a#12],,)
([a#12],,)
([a#12],,)
The Perl script used here is:
{code}
#!/usr/local/bin/perl
use warnings;
use strict;
while(<>) {
my($bc,$s,$m,$l)=split/\t/;
print("$s\t$m\t$l");
}
{code}
Is there an issue with handling of complex Map fields within streaming? How can I fix this to obtain the right result?
Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1243) Passing Complex map types to and from
streaming causes a problem
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-1243:
-----------------------------------
Assignee: Richard Ding
Richard, could you please take a look at what the issue is and whether there are workaround
> Passing Complex map types to and from streaming causes a problem
> ----------------------------------------------------------------
>
> Key: PIG-1243
> URL: https://issues.apache.org/jira/browse/PIG-1243
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
>
> I have a program which generates different types of Maps fields and stores it into PigStorage.
> {code}
> A = load '/user/viraj/three.txt' using PigStorage();
> B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}] as d;
> store B into '/user/viraj/pigtest' using PigStorage();
> {code}
> Now I test the previous output in the below script to make sure I have the right results. I also pass this data to a Perl script and I observe that the complex Map types I have generated, are lost when I get the result back.
> {code}
> DEFINE CMD `simple.pl` SHIP('simple.pl');
> A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);
> B = foreach A generate $0, $1, $2;
> dump B;
> C = foreach A generate (chararray)simpleFields#'a' as value, $0,$1,$2;
> D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);
> dump D;
> {code}
> dumping B results in:
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> dumping D results in:
> ([a#12],,)
> ([a#12],,)
> ([a#12],,)
> The Perl script used here is:
> {code}
> #!/usr/local/bin/perl
> use warnings;
> use strict;
> while(<>) {
> my($bc,$s,$m,$l)=split/\t/;
> print("$s\t$m\t$l");
> }
> {code}
> Is there an issue with handling of complex Map fields within streaming? How can I fix this to obtain the right result?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1243) Passing Complex map types to and from
streaming causes a problem
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding resolved PIG-1243.
-------------------------------
Resolution: Fixed
This is fixed by way of PIG-613.
> Passing Complex map types to and from streaming causes a problem
> ----------------------------------------------------------------
>
> Key: PIG-1243
> URL: https://issues.apache.org/jira/browse/PIG-1243
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
>
> I have a program which generates different types of Maps fields and stores it into PigStorage.
> {code}
> A = load '/user/viraj/three.txt' using PigStorage();
> B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}] as d;
> store B into '/user/viraj/pigtest' using PigStorage();
> {code}
> Now I test the previous output in the below script to make sure I have the right results. I also pass this data to a Perl script and I observe that the complex Map types I have generated, are lost when I get the result back.
> {code}
> DEFINE CMD `simple.pl` SHIP('simple.pl');
> A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);
> B = foreach A generate $0, $1, $2;
> dump B;
> C = foreach A generate (chararray)simpleFields#'a' as value, $0,$1,$2;
> D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);
> dump D;
> {code}
> dumping B results in:
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> dumping D results in:
> ([a#12],,)
> ([a#12],,)
> ([a#12],,)
> The Perl script used here is:
> {code}
> #!/usr/local/bin/perl
> use warnings;
> use strict;
> while(<>) {
> my($bc,$s,$m,$l)=split/\t/;
> print("$s\t$m\t$l");
> }
> {code}
> Is there an issue with handling of complex Map fields within streaming? How can I fix this to obtain the right result?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1243) Passing Complex map types to and from
streaming causes a problem
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1243.
---------------------------
> Passing Complex map types to and from streaming causes a problem
> ----------------------------------------------------------------
>
> Key: PIG-1243
> URL: https://issues.apache.org/jira/browse/PIG-1243
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Viraj Bhat
> Assignee: Richard Ding
> Fix For: 0.7.0
>
>
> I have a program which generates different types of Maps fields and stores it into PigStorage.
> {code}
> A = load '/user/viraj/three.txt' using PigStorage();
> B = foreach A generate ['a'#'12'] as b:map[], ['b'#['c'#'12']] as c, ['c'#{(['d'#'15']),(['e'#'16'])}] as d;
> store B into '/user/viraj/pigtest' using PigStorage();
> {code}
> Now I test the previous output in the below script to make sure I have the right results. I also pass this data to a Perl script and I observe that the complex Map types I have generated, are lost when I get the result back.
> {code}
> DEFINE CMD `simple.pl` SHIP('simple.pl');
> A = load '/user/viraj/pigtest' using PigStorage() as (simpleFields, mapFields, mapListFields);
> B = foreach A generate $0, $1, $2;
> dump B;
> C = foreach A generate (chararray)simpleFields#'a' as value, $0,$1,$2;
> D = stream C through CMD as (a0:map[], a1:map[], a2:map[]);
> dump D;
> {code}
> dumping B results in:
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> ([a#12],[b#[c#12]],[c#{([d#15]),([e#16])}])
> dumping D results in:
> ([a#12],,)
> ([a#12],,)
> ([a#12],,)
> The Perl script used here is:
> {code}
> #!/usr/local/bin/perl
> use warnings;
> use strict;
> while(<>) {
> my($bc,$s,$m,$l)=split/\t/;
> print("$s\t$m\t$l");
> }
> {code}
> Is there an issue with handling of complex Map fields within streaming? How can I fix this to obtain the right result?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.