You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2008/05/19 19:59:56 UTC
[jira] Created: (PIG-243) Make pig work on Windows
Make pig work on Windows
------------------------
Key: PIG-243
URL: https://issues.apache.org/jira/browse/PIG-243
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich
Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/30/08 10:23 AM:
----------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing '\r' (0x0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing '\r' |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598128#action_12598128 ]
Pi Song commented on PIG-243:
-----------------------------
Like Alan said, our developers don't use Windows so it's difficult to test. Is it possible to also run tests on windows as nightly build?
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-243:
---------------------------
Attachment: PIG_243.patch
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/28/08 12:54 AM:
------------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing need further work. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/28/08 12:52 AM:
------------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing need further work. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi, |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Patch for PIG under cygwin
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jianyong Dai updated PIG-243:
-----------------------------
Status: Patch Available (was: Open)
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600813#action_12600813 ]
Olga Natkovich commented on PIG-243:
------------------------------------
Thanks, Daniel. Changes look good. Once concern I have is with changes to PigStorage since it adds a little bit of overhead to each record processing. However, since I assume that String caches the length, the overhead should be minimal. Could you run a quick test that reads a million lines with and without the extra processing and see what the time difference is. Thanks.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/28/08 9:03 AM:
-----------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-243:
---------------------------
Attachment: cygpath.patch
Fix the remaining 4 unit test error under cygwin
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: cygpath.patch, PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601242#action_12601242 ]
Olga Natkovich commented on PIG-243:
------------------------------------
Patch committed. Thanks Daniel for contributing!
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/28/08 12:53 AM:
------------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing need further work. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing need further work. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi, |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " |
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Jianyong Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jianyong Dai updated PIG-243:
-----------------------------
Attachment: PIG_243.patch
Patch for PIG under cygwin
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600905#action_12600905 ]
Daniel Dai commented on PIG-243:
--------------------------------
I have tested reading one million records. Here is the result:
* On Linux, where trimming not taking place:
** Before patch: 10.53s
** After patch: 10.60s
* On Windows, where trimming branch taken:
** Before patch: 13.65s
** After patch: 13.69s
Seems that the overhead is neglectable.
Another solution is to change the testcase rather than core code to fix this test failure. However, I think move it to core is better. Failing to read a file in dos format under cygwin sounds not reasonable. How do you think?
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601104#action_12601104 ]
Pi Song commented on PIG-243:
-----------------------------
+1
BTW, there is another guy called TextLoader that needs '\r' removed as well.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/30/08 11:26 AM:
----------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing '\r' (0x0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing '\r' |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing '\r') is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing '\r' (0x0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing '\r' |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608231#action_12608231 ]
Olga Natkovich commented on PIG-243:
------------------------------------
Thanks Daniel. Running tests now.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: cygpath.patch, PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-243:
---------------------------
Attachment: (was: PIG_243.patch)
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 6/24/08 9:56 AM:
---------------------------------------------------------
Attached patch fix -most- all problems under cygwin. -4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work.- Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing '\r' (0x0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing '\r' |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
| Pig streaming put the current directory into PATH. Current directory is obtained using "System.getProperty("user.dir")", it is a Windows style path under cygwin. However, Cygwin only accept Unix style path in PATH environment variable. So put the Windows path obtained from "System.getProperty" directly into PATH will not work | TestStreaming | Change Windows path to Unix path under Cygwin. Using Cygwin utility "cygpath"|
Only two change are in core code. One is PigStorage.java (to fix tailing '\r'). The other is ExecutableManager (Add cygpath). All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing '\r' (0x0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing '\r' |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing '\r') is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: cygpath.patch, PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601213#action_12601213 ]
Daniel Dai commented on PIG-243:
--------------------------------
You are right. We should change TextLoader as well. I have updated patch to include this. Thank you!
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600925#action_12600925 ]
Olga Natkovich commented on PIG-243:
------------------------------------
Yes, I think this is fine. I will test and commit the patch sometimes tomorrow or Monday.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600384#action_12600384 ]
daijy edited comment on PIG-243 at 5/29/08 2:36 PM:
---------------------------------------------------------
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\ \", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
was (Author: daijy):
Attached patch fix most problems under cygwin. 4 errors remaining in TestStreaming for map-reduce testing under cygwin need further work. Patch is tested in both Linux and Cygwin. Here is a list of detected problem and solution:
||Problem||Affected Testcase||Solution||
|Windows path delimit "\" interpreted as escape in java|TestMapReduceResultRecycling, TestMapReduce, TestOrderBy, TestBuiltin, TestFilterOpNumeric, TestBinaryStorage, TestFilterOpString, TestEvalPipeline, TestPigFile, TestReversibleLoadStore, TestAlgebraicEval, TestLargeFile, TestCombiner, TestPi |replace "\" with "\\", see [http://issues.apache.org/jira/browse/PIG-152]|
| Runtime.exec(String[]) fail for perl -e if perl statement include quote character for unknown reason |TestParamSubPreproc, TestStreaming|append a escape character \ to quote character " under Windows|
| There is a tailing CR (ascii: 0d) character for line read from text file under cygwin|TestBuiltin, TestEvalPipeline, TestPigSplit, TestStreaming|remove tailing 0x0d |
| Bug: forget to close output stream, following deletion operation fail|TestBinaryStorage|Fix bug |
Only one change to PigStorage.java (to fix tailing 0x0d) is inside core code. All other changes are in testcase.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-243) Make pig work on Windows
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-243:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed. Thanks Daniel for contributing.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: cygpath.patch, PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601602#action_12601602 ]
Pi Song commented on PIG-243:
-----------------------------
So far, we have ensured the data internalize/externalize bit. Is there anything else in the engine that can cause trouble on Windows?
I couldn't think of any.
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600931#action_12600931 ]
Daniel Dai commented on PIG-243:
--------------------------------
Thank you!
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-243) Make pig work on Windows
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600905#action_12600905 ]
daijy edited comment on PIG-243 at 5/29/08 1:19 PM:
---------------------------------------------------------
I have tested reading one million records. Here is the result:
* On Linux, where trimming not taking place:
** Before patch: 10.53s
** After patch: 10.60s
* On Windows, where trimming branch taken:
** Before patch: 13.65s
** After patch: 13.69s
Seems that the overhead is neglectable.
Another solution is to change the testcase rather than core code to fix this test failure. However, I think move it to core is better. Failing to read a file in dos format (or need extra processing) under cygwin sounds not reasonable. How do you think?
was (Author: daijy):
I have tested reading one million records. Here is the result:
* On Linux, where trimming not taking place:
** Before patch: 10.53s
** After patch: 10.60s
* On Windows, where trimming branch taken:
** Before patch: 13.65s
** After patch: 13.69s
Seems that the overhead is neglectable.
Another solution is to change the testcase rather than core code to fix this test failure. However, I think move it to core is better. Failing to read a file in dos format under cygwin sounds not reasonable. How do you think?
> Make pig work on Windows
> ------------------------
>
> Key: PIG-243
> URL: https://issues.apache.org/jira/browse/PIG-243
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Attachments: PIG_243.patch
>
>
> Currently a large number of unit tests is failing on Windows. We need to fix that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.