You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "hc busy (JIRA)" <ji...@apache.org> on 2010/04/21 07:12:49 UTC
[jira] Created: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
UDF to extend functionalities of MaxTupleBy1stField
---------------------------------------------------
Key: PIG-1386
URL: https://issues.apache.org/jira/browse/PIG-1386
Project: Pig
Issue Type: New Feature
Components: tools
Affects Versions: 0.6.0
Reporter: hc busy
Based on this conversation:
totally, go for it, it'd be pretty straightforward to add this
functionality.
- Hide quoted text -
On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> Hey, while we're on the subject, and I have your attention, can we
> re-factor
> the UDF MaxTupleByFirstField to take constructor?
>
> *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> *G = group T by id;*
> *M = foreach T generate customMaxTuple(T);
> *
>
> Where n is the nth field, and the second parameter allows us to specify
> "min", "max", "median", etc...
>
> Does this seem like something useful to everyone?
>
>
>
> On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
>
> > What about making them part of the language using symbols?
> >
> > instead of
> >
> > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> >
> > have language support
> >
> > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> >
> > or even:
> >
> > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> >
> >
> > Is there reason not to do the second or third other than being more
> > complicated?
> >
> > Certainly I'd volunteer to put the top implementation in to the util
> > package and submit them for builtin's, but the latter syntactic candies
> > seems more natural..
> >
> >
> >
> > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> >
> >> The grouping package in piggybank is left over from back when Pig
> allowed
> >> users to define grouping functions (0.1). Functions like these should
> go in
> >> evaluation.util.
> >>
> >> However, I'd consider putting these in builtin (in main Pig) instead.
> >> These are things everyone asks for and they seem like a reasonable
> addition
> >> to the core engine. This will be more of a burden to write (as we'll
> hold
> >> them to a higher standard) but of more use to people as well.
> >>
> >> Alan.
> >>
> >>
> >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> >>
> >> Some times I wonder... I mean, somebody went to the trouble of making a
> >>> path
> >>> called
> >>>
> >>> org.apache.pig.piggybank.grouping
> >>>
> >>> (where it seems like this code belong), but didn't check in any java
> code
> >>> into that package.
> >>>
> >>>
> >>> Any comment about where to put this kind of utility classes?
> >>>
> >>>
> >>>
> >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> >>>
> >>> 2010/4/19 hc busy <hc...@gmail.com>
> >>>>
> >>>> That's just the way it is right now, you can't make bags or tuples
> >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> >>>>>
> >>>>> toBag()
> >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> >>>>> TupleToBag(); --some times you need it this way for some reason.
> >>>>>
> >>>>>
> >>>>> Ok. I place my current code here, may be later I make a patch (if
> such
> >>>> implementation is acceptable of course).
> >>>>
> >>>> import org.apache.pig.EvalFunc;
> >>>> import org.apache.pig.data.BagFactory;
> >>>> import org.apache.pig.data.DataBag;
> >>>> import org.apache.pig.data.Tuple;
> >>>> import org.apache.pig.data.TupleFactory;
> >>>>
> >>>> import java.io.IOException;
> >>>>
> >>>> /**
> >>>> * Convert any sequence of fields to bag with specified count of
> >>>> fields<br>
> >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> >>>> *
> >>>> * @author astepachev
> >>>> */
> >>>> public class ToBag extends EvalFunc<DataBag> {
> >>>> public BagFactory bagFactory;
> >>>> public TupleFactory tupleFactory;
> >>>>
> >>>> public ToBag() {
> >>>> bagFactory = BagFactory.getInstance();
> >>>> tupleFactory = TupleFactory.getInstance();
> >>>> }
> >>>>
> >>>> @Override
> >>>> public DataBag exec(Tuple input) throws IOException {
> >>>> if (input.isNull())
> >>>> return null;
> >>>> final DataBag bag = bagFactory.newDefaultBag();
> >>>> final Integer couter = (Integer) input.get(0);
> >>>> if (couter == null)
> >>>> return null;
> >>>> Tuple tuple = tupleFactory.newTuple();
> >>>> for (int i = 0; i < input.size() - 1; i++) {
> >>>> if (i % couter == 0) {
> >>>> tuple = tupleFactory.newTuple();
> >>>> bag.add(tuple);
> >>>> }
> >>>> tuple.append(input.get(i + 1));
> >>>> }
> >>>> return bag;
> >>>> }
> >>>> }
> >>>>
> >>>> import org.apache.pig.ExecType;
> >>>> import org.apache.pig.PigServer;
> >>>> import org.junit.Before;
> >>>> import org.junit.Test;
> >>>>
> >>>> import java.io.IOException;
> >>>> import java.net.URISyntaxException;
> >>>> import java.net.URL;
> >>>>
> >>>> import static org.junit.Assert.assertTrue;
> >>>>
> >>>> /**
> >>>> * @author astepachev
> >>>> */
> >>>> public class ToBagTest {
> >>>> PigServer pigServer;
> >>>> URL inputTxt;
> >>>>
> >>>> @Before
> >>>> public void init() throws IOException, URISyntaxException {
> >>>> pigServer = new PigServer(ExecType.LOCAL);
> >>>> inputTxt =
> >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> >>>> }
> >>>>
> >>>> @Test
> >>>> public void testSimple() throws IOException {
> >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> +
> >>>> "' using PigStorage(',') " +
> >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> >>>> d:chararray);");
> >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> >>>>
> >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> >>>> pigServer.store("last", "target/pigtest/func1.txt");
> >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> >>>> }
> >>>> }
> >>>>
> >>>>
> >>
> >
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
The patch
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860121#action_12860121 ]
Hadoop QA commented on PIG-1386:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442629/PIG-1386-trunk.patch
against trunk revision 937095.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 530 release audit warnings (more than the trunk's current 528 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/301/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/301/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/301/console
This message is automatically generated.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1386:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
patch committed. Thanks hc busy!
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Open (was: Patch Available)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Patch Available (was: Open)
Here's a first stab.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Open (was: Patch Available)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
a92218b0c641363439af8f2d9e5ecbc0
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
e503949c4f5f2667657ee02872aff5ce
Additional documentation and examples.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Patch Available (was: Open)
resubmitting patch for the build system to check.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Patch Available (was: Open)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
da673ab2d584faf903e8b49b63a03ade
spell check the documentation
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861544#action_12861544 ]
Hadoop QA commented on PIG-1386:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442926/PIG-1386-trunk.patch
against trunk revision 937570.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 537 release audit warnings (more than the trunk's current 535 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/305/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/305/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/305/console
This message is automatically generated.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
163812d67299dd4b44470c854c80f2a8
redo without the addition of the helper function.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reassigned PIG-1386:
-------------------------------
Assignee: hc busy
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859291#action_12859291 ]
Hadoop QA commented on PIG-1386:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442396/PIG-1386-trunk.patch
against trunk revision 935968.
-1 @author. The patch appears to contain 1 @author tags which the Pig community has agreed to not allow in code contributions.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 537 release audit warnings (more than the trunk's current 535 warnings).
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/297/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/297/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/297/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/297/console
This message is automatically generated.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860104#action_12860104 ]
hc busy commented on PIG-1386:
------------------------------
I'm having trouble writing this UDF because of the bug similar to PIG-1303; Here's my comment to that ticket below. It seems that by doing this, it allows me to pass on the constructor parameters:
{quote}
Okay, so, here's a thought:
I'm kind of stuck writing the initial/intermed/Final methods for an algebraic EvalFunc that has constructor parameters because I couldn't pass the parameters in.
A suggestion is to do this (without being incompatible with previous versions)
Alter EvalFunc's profile so that
{code}
public abstract class EvalFunc<T> {
protected handleChildConstructorParameters(Object... childConstructor){
// by default do nothing.
}
public EvalFunc(Object... constructorParameters){
handleChildConstructorParameters(constructorParameters);
... then do everything else it used to do.
}
}
{code}
The reason why this is necessary is because I'll need to overrite handleChildConstructorParameters in my Algebraic EvalFunc to do some things before the rest of EvalFunc()'s constructor continues. This will help fix this date format problem for Algebraic evalfunc's.
{quote}
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
checked to be sure the unittest builds and runs.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
1873fb8d75f7362df343615f623a7390
Added documentation, added a bunch of unit tests to test the functionalities that the documentation claims to have. cleaned up to revert to not requiring change to EvalFunc's constructor. Added ASF license text.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Patch Available (was: Open)
Fix Version/s: 0.8.0
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861588#action_12861588 ]
Hadoop QA commented on PIG-1386:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442973/PIG-1386-trunk.patch
against trunk revision 937570.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/303/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/303/console
This message is automatically generated.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Status: Open (was: Patch Available)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860490#action_12860490 ]
hc busy commented on PIG-1386:
------------------------------
Okay, here's an alternative. What if we did this instead:
{code}
class EvalFunc{
...
protected String parameters="";
public EvalFunc(Object... constructorParameters){
StringBuilder sb = new StringBuilder();
if(constructorParameters!=null && constructorParameters.length>0){
for(Object o:constructorParameters){
sb.append(',');
sb.append('\'');
sb.append(o.toString());
sb.append('\'');}
parameters="("+sb.substring(1)+")";
}
... //rest of evalfunc constructor.
...
}
{code}
and my getInitial is implemented thusly:
{code}
@Override
public String getInitial() {
return HelperClass.class.getName() + parameters;
}
{code}
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894720#action_12894720 ]
Olga Natkovich commented on PIG-1386:
-------------------------------------
Is this ready to be committed?
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Fix For: 0.8.0
>
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860325#action_12860325 ]
hc busy commented on PIG-1386:
------------------------------
oops, posted to the wrong ticket:
{quote}
Hmm, okay, so let me shorten my problem. Basically the functions getInitial(), getIntermed(), and getFinal() in my Algebraic class doesn't have access to the constructor parameters. The reason is this. in Java, the super() constructor can only be called as the very first thing that the deriving class's constructor does, so my udfs has constructors that look like this:
{code}
public ExtremalTupleByNthField(String fieldIndexString, String order) {
super();
parameters = "('"+fieldIndexString+"','"+order+"'";
}
@Override
public String getInitial() {
return HelperClass.class.getName()+parameters;
}
{code}
But the problem is EvalFunc() constructor initializes the EvalFunc as returned by getInitial() to type check. When it does this, it finds that my getInitial() returns something incomplete because the "parameters" member variable hasn't been initialized yet. This is a pretty mundane problem with java programs and the way to fix it is what I've submitted in the patch calling an overridden method in the super()'s constructor.
I mean, I don't see any other way to do this, but I'd be willing to work on another implementation if you can suggest one?
{quote}
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: PIG-1386-trunk.patch
25ce97367cadfd2ea4be379c6f5c351d
Clean up documentation and refactor to unify parsing of constructor arguments in the two classes.
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1386) UDF to extend functionalities of
MaxTupleBy1stField
Posted by "hc busy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hc busy updated PIG-1386:
-------------------------
Attachment: (was: PIG-1386-trunk.patch)
> UDF to extend functionalities of MaxTupleBy1stField
> ---------------------------------------------------
>
> Key: PIG-1386
> URL: https://issues.apache.org/jira/browse/PIG-1386
> Project: Pig
> Issue Type: New Feature
> Components: tools
> Affects Versions: 0.6.0
> Reporter: hc busy
> Assignee: hc busy
> Attachments: PIG-1386-trunk.patch
>
>
> Based on this conversation:
> totally, go for it, it'd be pretty straightforward to add this
> functionality.
> - Hide quoted text -
> On Tue, Apr 20, 2010 at 6:45 PM, hc busy <hc...@gmail.com> wrote:
> > Hey, while we're on the subject, and I have your attention, can we
> > re-factor
> > the UDF MaxTupleByFirstField to take constructor?
> >
> > *define customMaxTuple ExtremalTupleByNthField(n, 'min');*
> > *G = group T by id;*
> > *M = foreach T generate customMaxTuple(T);
> > *
> >
> > Where n is the nth field, and the second parameter allows us to specify
> > "min", "max", "median", etc...
> >
> > Does this seem like something useful to everyone?
> >
> >
> >
> > On Tue, Apr 20, 2010 at 6:34 PM, hc busy <hc...@gmail.com> wrote:
> >
> > > What about making them part of the language using symbols?
> > >
> > > instead of
> > >
> > > foreach T generate Tuple($0, $1, $2), Bag($3, $4, $5), $6, $7;
> > >
> > > have language support
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, $6, $7;
> > >
> > > or even:
> > >
> > > foreach T generate ($0, $1, $2), {$3, $4, $5}, [$6#$7, $8#$9], $10, $11;
> > >
> > >
> > > Is there reason not to do the second or third other than being more
> > > complicated?
> > >
> > > Certainly I'd volunteer to put the top implementation in to the util
> > > package and submit them for builtin's, but the latter syntactic candies
> > > seems more natural..
> > >
> > >
> > >
> > > On Tue, Apr 20, 2010 at 5:24 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > >
> > >> The grouping package in piggybank is left over from back when Pig
> > allowed
> > >> users to define grouping functions (0.1). Functions like these should
> > go in
> > >> evaluation.util.
> > >>
> > >> However, I'd consider putting these in builtin (in main Pig) instead.
> > >> These are things everyone asks for and they seem like a reasonable
> > addition
> > >> to the core engine. This will be more of a burden to write (as we'll
> > hold
> > >> them to a higher standard) but of more use to people as well.
> > >>
> > >> Alan.
> > >>
> > >>
> > >> On Apr 19, 2010, at 12:53 PM, hc busy wrote:
> > >>
> > >> Some times I wonder... I mean, somebody went to the trouble of making a
> > >>> path
> > >>> called
> > >>>
> > >>> org.apache.pig.piggybank.grouping
> > >>>
> > >>> (where it seems like this code belong), but didn't check in any java
> > code
> > >>> into that package.
> > >>>
> > >>>
> > >>> Any comment about where to put this kind of utility classes?
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Apr 19, 2010 at 12:07 PM, Andrey S <oc...@gmail.com> wrote:
> > >>>
> > >>> 2010/4/19 hc busy <hc...@gmail.com>
> > >>>>
> > >>>> That's just the way it is right now, you can't make bags or tuples
> > >>>>> directly... Maybe we should have some UDF's in piggybank for these:
> > >>>>>
> > >>>>> toBag()
> > >>>>> toTuple(); --which is kinda like exec(Tuple in){return in;}
> > >>>>> TupleToBag(); --some times you need it this way for some reason.
> > >>>>>
> > >>>>>
> > >>>>> Ok. I place my current code here, may be later I make a patch (if
> > such
> > >>>> implementation is acceptable of course).
> > >>>>
> > >>>> import org.apache.pig.EvalFunc;
> > >>>> import org.apache.pig.data.BagFactory;
> > >>>> import org.apache.pig.data.DataBag;
> > >>>> import org.apache.pig.data.Tuple;
> > >>>> import org.apache.pig.data.TupleFactory;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>>
> > >>>> /**
> > >>>> * Convert any sequence of fields to bag with specified count of
> > >>>> fields<br>
> > >>>> * Schema: count:int, fld1 [, fld2, fld3, fld4... ].
> > >>>> * Output: count=2, then { (fld1, fld2) , (fld3, fld4) ... }
> > >>>> *
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBag extends EvalFunc<DataBag> {
> > >>>> public BagFactory bagFactory;
> > >>>> public TupleFactory tupleFactory;
> > >>>>
> > >>>> public ToBag() {
> > >>>> bagFactory = BagFactory.getInstance();
> > >>>> tupleFactory = TupleFactory.getInstance();
> > >>>> }
> > >>>>
> > >>>> @Override
> > >>>> public DataBag exec(Tuple input) throws IOException {
> > >>>> if (input.isNull())
> > >>>> return null;
> > >>>> final DataBag bag = bagFactory.newDefaultBag();
> > >>>> final Integer couter = (Integer) input.get(0);
> > >>>> if (couter == null)
> > >>>> return null;
> > >>>> Tuple tuple = tupleFactory.newTuple();
> > >>>> for (int i = 0; i < input.size() - 1; i++) {
> > >>>> if (i % couter == 0) {
> > >>>> tuple = tupleFactory.newTuple();
> > >>>> bag.add(tuple);
> > >>>> }
> > >>>> tuple.append(input.get(i + 1));
> > >>>> }
> > >>>> return bag;
> > >>>> }
> > >>>> }
> > >>>>
> > >>>> import org.apache.pig.ExecType;
> > >>>> import org.apache.pig.PigServer;
> > >>>> import org.junit.Before;
> > >>>> import org.junit.Test;
> > >>>>
> > >>>> import java.io.IOException;
> > >>>> import java.net.URISyntaxException;
> > >>>> import java.net.URL;
> > >>>>
> > >>>> import static org.junit.Assert.assertTrue;
> > >>>>
> > >>>> /**
> > >>>> * @author astepachev
> > >>>> */
> > >>>> public class ToBagTest {
> > >>>> PigServer pigServer;
> > >>>> URL inputTxt;
> > >>>>
> > >>>> @Before
> > >>>> public void init() throws IOException, URISyntaxException {
> > >>>> pigServer = new PigServer(ExecType.LOCAL);
> > >>>> inputTxt =
> > >>>> this.getClass().getResource("bagTest.txt").toURI().toURL();
> > >>>> }
> > >>>>
> > >>>> @Test
> > >>>> public void testSimple() throws IOException {
> > >>>> pigServer.registerQuery("a = load '" + inputTxt.toExternalForm()
> > +
> > >>>> "' using PigStorage(',') " +
> > >>>> "as (id:int, a:chararray, b:chararray, c:chararray,
> > >>>> d:chararray);");
> > >>>> pigServer.registerQuery("last = foreach a generate flatten(" +
> > >>>> ToBag.class.getName() + "(2, id, a, id, b, id, c));");
> > >>>>
> > >>>> pigServer.deleteFile("target/pigtest/func1.txt");
> > >>>> pigServer.store("last", "target/pigtest/func1.txt");
> > >>>> assertTrue(pigServer.fileSize("target/pigtest/func1.txt") > 0);
> > >>>> }
> > >>>> }
> > >>>>
> > >>>>
> > >>
> > >
> >
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.