You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Xu Zhang (JIRA)" <ji...@apache.org> on 2008/03/17 21:34:24 UTC
[jira] Created: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
--------------------------------------------------------------------------------------------
Key: PIG-152
URL: https://issues.apache.org/jira/browse/PIG-152
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Xu Zhang
*Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
{noformat}
org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
Standard Output
Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
{noformat}
Here is the corresponding code:
{code}
public class TestAlgebraicEval extends TestCase {
MiniCluster cluster = MiniCluster.buildCluster();
@Test
public void testSimpleCount() throws Throwable {
int LOOP_COUNT = 1024;
PigServer pig = new PigServer(MAPREDUCE);
File tmpFile = File.createTempFile("test", "txt");
PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
for(int i = 0; i < LOOP_COUNT; i++) {
ps.println(i);
}
ps.close();
String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
System.out.println(query);
pig.registerQuery(query);
Iterator it = pig.openIterator("myid");
tmpFile.delete();
Tuple t = (Tuple)it.next();
Double count = t.getAtomField(0).numval();
assertEquals(count, (double)LOOP_COUNT);
}
.
.
.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579710#action_12579710 ]
Xu Zhang commented on PIG-152:
------------------------------
Ok, I have found the solution (or a workaround?). It took me almost whole day to get the thing to work on a remote Windows machine with the dreaded Strng::replaceAll() method, where "\" is also used for escape.
However, before I go ahead adding code to every test method of all the test case classes where the query string contains "\", I would like to pose the following questions:
*Is this an issue in the unit test code, or in Pig itself?* What if the user issues the same commands via grunt on a Windows machine? In that case, should they be required to escape "\" as well?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song reassigned PIG-152:
---------------------------
Assignee: Pi Song
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579666#action_12579666 ]
Pi Song commented on PIG-152:
-----------------------------
Sounds right to me.
I forgot to change other unit tests to use the new semantic. Sorry for that.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579622#action_12579622 ]
Xu Zhang commented on PIG-152:
------------------------------
Thanks Pi. I do not think there is any special tricks besides those with
Linux (such as you will need ant, junit, java, svn, etc). If you want, you
can also send me your pig.jar and then I will try it out for you.
Best regards,
Xu
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373>>
)
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580165#action_12580165 ]
Pi Song commented on PIG-152:
-----------------------------
I already reviewed. It is ready to be committed.
Please make sure all the unit tests work.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579890#action_12579890 ]
Xu Zhang commented on PIG-152:
------------------------------
Pi: What you said makes sense to me. I will work on the unit test code as what you said.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-152) Solution for PIG-123 might
have introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579642#action_12579642 ]
xuzh edited comment on PIG-152 at 3/17/08 3:22 PM:
-------------------------------------------------------
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\\\" in the unit tests code sound to everyone?
was (Author: xuzh):
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\" in the unit tests code sound to everyone?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-152) Solution for PIG-123 might
have introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579642#action_12579642 ]
xuzh edited comment on PIG-152 at 3/17/08 3:22 PM:
-------------------------------------------------------
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\\" in the unit tests code sound to everyone?
was (Author: xuzh):
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\\\" in the unit tests code sound to everyone?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-152) Solution for PIG-123 might
have introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579642#action_12579642 ]
xuzh edited comment on PIG-152 at 3/17/08 3:23 PM:
-------------------------------------------------------
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\" in the unit tests code sound to everyone?
was (Author: xuzh):
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\\" in the unit tests code sound to everyone?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580167#action_12580167 ]
Xu Zhang commented on PIG-152:
------------------------------
Tested already before submitting the patch.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Zhang updated PIG-152:
-------------------------
Attachment: PIG_152.patch
Attach the patch. Changes include adding the static method encodeEscape() to Util.java and calling this method in the test classes to replace "\" with "\\".
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579844#action_12579844 ]
Pi Song commented on PIG-152:
-----------------------------
Xu Zhang:
{quote}
should they be required to escape "\" as well?
{quote}
To answer this question, I regard Pig as a developer tool like Java or SQL so adding a bit more complexity to support more use cases is acceptable. In both Java and SQL you always have to know about escape characters.
Example from javac
{noformat}
Test1.java:3: illegal escape character
System.out.println("This is Mr.Pi \M hohoho");
^
1 error
{noformat}
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Zhang reassigned PIG-152:
----------------------------
Assignee: Xu Zhang (was: Pi Song)
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Xu Zhang
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579624#action_12579624 ]
Pi Song commented on PIG-152:
-----------------------------
Xu,
Please try changing all "\" to "\ \" like what I said.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Zhang updated PIG-152:
-------------------------
Status: Patch Available (was: Open)
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579840#action_12579840 ]
Pi Song commented on PIG-152:
-----------------------------
Xu,
Please read PIG-123 again.
We've changed the string input semantic from "\" being a normal char to "\" being an escape char. Therefore from now on if you want to refer to "\" itself you will always have to use double "\". This will affect both Grunt and all unit tests.
In fact, this kind of breaking changes are supposed to be highlighted in the changelog but I don't even see it!!
Regarding the tests, I don't have a windows machine to test. Could you please encode all queries in unit tests to use double "\" for me?
Let's create TestUtil.EncodeEscape method to do that. One common scenario that you will have to encode is when we generate a temp file to use as an input.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-152:
------------------------
Patch Info: [Patch Available]
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Xu Zhang
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-152:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Opened generic windows issue: https://issues.apache.org/jira/browse/PIG-243
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Xu Zhang
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579642#action_12579642 ]
Xu Zhang commented on PIG-152:
------------------------------
Continued from my previous comment: You might also need Cygwin for running Pig on Windows. This is because Hadoop needs Cygwin to run on Windows.
For Pi's latest comment, I will try hard coding the temp file path to see if that fixes the issue. However, we will need to find a permanent solution for this. How does fixing by scanning the path string and replacing all instances of "\" with "\\" in the unit tests code sound to everyone?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by Xu Zhang <xu...@yahoo-inc.com>.
Thanks Pi. I do not think there is any special tricks besides those with
Linux (such as you will need ant, junit, java, svn, etc). If you want, you
can also send me your pig.jar and then I will try it out for you.
Best regards,
Xu
> From: "Pi Song (JIRA)" <ji...@apache.org>
> Reply-To: <pi...@incubator.apache.org>
> Date: Mon, 17 Mar 2008 14:12:24 -0700 (PDT)
> To: <pi...@incubator.apache.org>
> Subject: [jira] Commented: (PIG-152) Solution for PIG-123 might have
> introduced parsing errors on Windows with the Pig unit tests
>
>
> [
> https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.s
> ystem.issuetabpanels:comment-tabpanel&focusedCommentId=12579606#action_1257960
> 6 ]
>
> Pi Song commented on PIG-152:
> -----------------------------
>
> Xu, I'll be looking at this as soon as possible. In the mean time, can you
> tell me if there is any trick to get Pig running on Windows?
>
>> Solution for PIG-123 might have introduced parsing errors on Windows with the
>> Pig unit tests
>> -----------------------------------------------------------------------------
>> ---------------
>>
>> Key: PIG-152
>> URL: https://issues.apache.org/jira/browse/PIG-152
>> Project: Pig
>> Issue Type: Bug
>> Components: impl
>> Reporter: Xu Zhang
>>
>> *Since the changes for PIG-123 were checked in, we got lots of parsing errors
>> when running the unit tests on Windows.*
>> It seems all these errors are generated when a query string that contains a
>> single quote for the path of the data file is passed into the
>> PigServer::registerQuery() method.
>> Here is an example of such an error and the location where it is generated in
>> the unit test code (on the line "pig.registerQuery(query);" in the following
>> code):
>> {noformat}
>> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line
>> 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(
>> QueryParserTokenManager.java:1599)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryPar
>> ser.java:4069)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java
>> :717)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.ja
>> va:615)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :497)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.ja
>> va:425)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.jav
>> a:1043)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser
>> .java:996)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :523)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.ja
>> va:425)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser
>> .java:1364)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :552)
>> at
>>
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373>>
)
>> at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:22
>> 7)
>> at
>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.
>> java:47)
>> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
>> at
>> org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:
>> 50)
>> Standard Output
>> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
>> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
>> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
>> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all)
>> generate COUNT($1);
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all)
>> generate group, COUNT($1) ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all)
>> generate COUNT($1), group ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using
>> org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using
>> org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1),
>> COUNT($1.$0) ;
>> {noformat}
>> Here is the corresponding code:
>> {code}
>> public class TestAlgebraicEval extends TestCase {
>>
>> MiniCluster cluster = MiniCluster.buildCluster();
>> @Test
>> public void testSimpleCount() throws Throwable {
>> int LOOP_COUNT = 1024;
>> PigServer pig = new PigServer(MAPREDUCE);
>> File tmpFile = File.createTempFile("test", "txt");
>> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
>> for(int i = 0; i < LOOP_COUNT; i++) {
>> ps.println(i);
>> }
>> ps.close();
>> String query = "myid = foreach (group (load 'file:" + tmpFile + "')
>> all) generate COUNT($1);";
>> System.out.println(query);
>> pig.registerQuery(query);
>> Iterator it = pig.openIterator("myid");
>> tmpFile.delete();
>> Tuple t = (Tuple)it.next();
>> Double count = t.getAtomField(0).numval();
>> assertEquals(count, (double)LOOP_COUNT);
>> }
>> .
>> .
>> .
>> {code}
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579606#action_12579606 ]
Pi Song commented on PIG-152:
-----------------------------
Xu, I'll be looking at this as soon as possible. In the mean time, can you tell me if there is any trick to get Pig running on Windows?
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579610#action_12579610 ]
Pi Song commented on PIG-152:
-----------------------------
A minute, I think I know what the problem is. *This is due to the change in string input semantic*.
Look at these lines:-
{noformat}
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
{noformat}
In order to input "\", from now on you have to do "\ \".
So
{noformat}
myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
{noformat}
should become
{noformat}
myid = foreach (group (load 'file:c:\\WINDOWS\\TEMP\\test39837txt') all) generate COUNT($1); (Like in C++/Java)
{noformat}
In Linux we don't use "\" to separate path components so we don't have this problem.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-152) Solution for PIG-123 might
have introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580153#action_12580153 ]
xuzh edited comment on PIG-152 at 3/18/08 3:44 PM:
-------------------------------------------------------
Attach the patch. Changes include adding the static method encodeEscape() to Util.java and calling this method in the test classes to replace single "\" with double "\".
was (Author: xuzh):
Attach the patch. Changes include adding the static method encodeEscape() to Util.java and calling this method in the test classes to replace "\" with "\\".
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
> Assignee: Pi Song
> Attachments: PIG_152.patch
>
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-152) Solution for PIG-123 might have
introduced parsing errors on Windows with the Pig unit tests
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579623#action_12579623 ]
Xu Zhang commented on PIG-152:
------------------------------
Thanks Pi. I do not think there is any special tricks besides the same ones with Linux (such as you will need ant, junit, java, svn, etc). If you want, you can also send me your pig.jar and then I will try it out for you.
> Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
> --------------------------------------------------------------------------------------------
>
> Key: PIG-152
> URL: https://issues.apache.org/jira/browse/PIG-152
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Xu Zhang
>
> *Since the changes for PIG-123 were checked in, we got lots of parsing errors when running the unit tests on Windows.*
> It seems all these errors are generated when a query string that contains a single quote for the path of the data file is passed into the PigServer::registerQuery() method.
> Here is an example of such an error and the location where it is generated in the unit test code (on the line "pig.registerQuery(query);" in the following code):
> {noformat}
> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 39. Encountered: "W" (87), after : "\'file:c:\\"
> at org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(QueryParserTokenManager.java:1599)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryParser.java:4069)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java:717)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:615)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:497)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.java:1043)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:996)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:523)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.java:425)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:1364)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:552)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:227)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:47)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
> at org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:50)
> Standard Output
> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all) generate COUNT($1);
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all) generate group, COUNT($1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all) generate COUNT($1), group ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1), COUNT($1.$0) ;
> {noformat}
> Here is the corresponding code:
> {code}
> public class TestAlgebraicEval extends TestCase {
>
> MiniCluster cluster = MiniCluster.buildCluster();
> @Test
> public void testSimpleCount() throws Throwable {
> int LOOP_COUNT = 1024;
> PigServer pig = new PigServer(MAPREDUCE);
> File tmpFile = File.createTempFile("test", "txt");
> PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
> for(int i = 0; i < LOOP_COUNT; i++) {
> ps.println(i);
> }
> ps.close();
> String query = "myid = foreach (group (load 'file:" + tmpFile + "') all) generate COUNT($1);";
> System.out.println(query);
> pig.registerQuery(query);
> Iterator it = pig.openIterator("myid");
> tmpFile.delete();
> Tuple t = (Tuple)it.next();
> Double count = t.getAtomField(0).numval();
> assertEquals(count, (double)LOOP_COUNT);
> }
> .
> .
> .
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.