You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Xu Zhang (JIRA)" <ji...@apache.org> on 2008/04/05 05:15:24 UTC
[jira] Created: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
There seems to be some mismatches between the actual stderr log and what I expected
-----------------------------------------------------------------------------------
Key: PIG-188
URL: https://issues.apache.org/jira/browse/PIG-188
Project: Pig
Issue Type: Bug
Reporter: Xu Zhang
Assignee: Arun C Murthy
With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0182, there were a total of 26 tasks (6 maps and 20 reduces).
>From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
In addition, the Input-split *** values in the logs do not make much sense to me:
{quote}
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
{quote}
Here is Pig script:
{code}
define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
B = stream A through X;
store B into 'logging_test_1';
C = load 'logging_test_1/_logs/logging_test_1';
store C into 'results_26';
{code}
Here are the logs:
{noformat}
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:18:44 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:18:45 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:31:34 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:31:36 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:22 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:22 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:28 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:28 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:32 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:33 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:37 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:37 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Zhang updated PIG-188:
-------------------------
Description:
With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
>From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
In addition, the Input-split *** values in the logs do not make much sense to me:
{quote}
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
{quote}
Here is Pig script:
{code}
define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
B = stream A through X;
store B into 'logging_test_1';
C = load 'logging_test_1/_logs/logging_test_1';
store C into 'results_26';
{code}
Here are the logs:
{noformat}
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:18:44 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:18:45 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:31:34 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:31:36 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:22 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:22 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:28 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:28 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:32 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:33 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:37 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:37 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
{noformat}
was:
With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0182, there were a total of 26 tasks (6 maps and 20 reduces).
>From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
In addition, the Input-split *** values in the logs do not make much sense to me:
{quote}
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
{quote}
Here is Pig script:
{code}
define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
B = stream A through X;
store B into 'logging_test_1';
C = load 'logging_test_1/_logs/logging_test_1';
store C into 'results_26';
{code}
Here are the logs:
{noformat}
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:18:44 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:18:45 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: PigLoggingTest 10 t
Start time: Fri Apr 04 19:31:34 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
This is stderr message number 1
This is stderr message number 2
This is stderr message number 3
This is stderr message number 4
This is stderr message number 5
This is stderr message number 6
This is stderr message number 7
This is stderr message number 8
This is stderr message number 9
This is stderr message number 10
===== Task Information Footer =====
End time: Fri Apr 04 19:31:36 PDT 2008
Exit code: 0
Input records: 10000
Input bytes: 1898380 bytes
Output records: 4
Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:22 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:22 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:28 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:28 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:32 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:33 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
===== Task Information Header =====
Command: ./cplusplus/PigLoggingTest 10 t
Start time: Fri Apr 04 10:11:37 PDT 2008
Input-split file: null
Input-split start-offset: -1
Input-split length: -1
===== * * * =====
===== Task Information Footer =====
End time: Fri Apr 04 10:11:37 PDT 2008
Exit code: -127
Input records: 747
Input bytes: 141796 bytes
Output records: 0
Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
===== * * * =====
{noformat}
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-188:
------------------------------
Attachment: PIG-188_2_20080408.patch
Fixed for TestCustomSlicer too... my bad.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PIG-188_2_20080408.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-188) There seems to be some mismatches
between the actual stderr log and what I expected
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586865#action_12586865 ]
Olga Natkovich commented on PIG-188:
------------------------------------
Changes look good; however, the test for CustomSlicer is failing after I applied the patch:
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.SliceWrapper.makeReader(SliceWrapper.java:96)
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigInputFormat.getRecordReader(PigInputFormat.java:113)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:200)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:150)
08/04/08 16:00:29 INFO mapreduceExec.MapReduceLauncher: Pig progress = 0%
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-188:
------------------------------
Attachment: PIG-188_1_20080407.patch
Updated patch:
* Fix the break caused by PIG-55
* Ensures that we do not output information about input-splits for reduces in the stderr logs since it could confuse users...
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-188:
------------------------------
Status: Patch Available (was: Open)
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-188:
---------------------------
Resolution: Fixed
Fix Version/s: 0.1.0
Status: Resolved (was: Patch Available)
Fix checked in revision 647997. Thanks Arun.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Fix For: 0.1.0
>
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PIG-188_2_20080408.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-188) There seems to be some mismatches
between the actual stderr log and what I expected
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588790#action_12588790 ]
Benjamin Reed commented on PIG-188:
-----------------------------------
+1 looks good to me.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PIG-188_2_20080408.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-188) There seems to be some mismatches
between the actual stderr log and what I expected
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586263#action_12586263 ]
Arun C Murthy commented on PIG-188:
-----------------------------------
Xu,
1. How did u get 20 reduces and 1 map for your first job?
2. You should expect 21 logs (20maps and 1reduces) only on HDFS.
3. The null/-1/-1 data for input-splits is due to the fact that reduces work on map-outputs and not on HDFS data.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Xu Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xu Zhang updated PIG-188:
-------------------------
Attachment: PigLoggingTest.cpp
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0182, there were a total of 26 tasks (6 maps and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-188) There seems to be some mismatches
between the actual stderr log and what I expected
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588611#action_12588611 ]
Alan Gates commented on PIG-188:
--------------------------------
Changes look fine to me and all the tests pass.
I'd like to get input from Ben or Charlie Groves, both of whom worked on the split stuff this modifies, to make sure the changes fit well with that interface.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PIG-188_1_20080407.patch, PIG-188_2_20080408.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-188) There seems to be some mismatches between
the actual stderr log and what I expected
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-188:
------------------------------
Attachment: PIG-188_0_20080407.patch
Looks like PIG-55 broke the feature where the InputSplit was displayed correctly in the logs... fixed now.
> There seems to be some mismatches between the actual stderr log and what I expected
> -----------------------------------------------------------------------------------
>
> Key: PIG-188
> URL: https://issues.apache.org/jira/browse/PIG-188
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
> Attachments: PIG-188_0_20080407.patch, PigLoggingTest.cpp
>
>
> With the following Pig script, I got streaming logs as shown below. The job for running this script is job_200804041056_0182. What PigLoggingTest does in this case is simply take tab delimited lines from STDIN and then output them to SDTOUT as tab delimited lines (so the same line comes in and out of PigLogginTest) after spitting out 10 STDERR messages. Also as shown in the UI of job_200804041056_0181, there were a total of 21 tasks (1 map and 20 reduces).
> From all these, I would expect the number of input records and output records to match in the log. Also, I would expect there to be 26 logs. In addition, since there was no error when running the script, all exit code should 0.
> However, there are actually only 6 logs. The number of input records and output records does not match. The logs show that some of the tasks exit with -127.
> In addition, the Input-split *** values in the logs do not make much sense to me:
> {quote}
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> {quote}
> Here is Pig script:
> {code}
> define X `PigLoggingTest 10 t` ship('./cplusplus/PigLoggingTest') stderr('logging_test_1');
> A = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa);
> B = stream A through X;
> store B into 'logging_test_1';
> C = load 'logging_test_1/_logs/logging_test_1';
> store C into 'results_26';
> {code}
> Here are the logs:
> {noformat}
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:18:44 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:18:45 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: PigLoggingTest 10 t
> Start time: Fri Apr 04 19:31:34 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> This is stderr message number 1
> This is stderr message number 2
> This is stderr message number 3
> This is stderr message number 4
> This is stderr message number 5
> This is stderr message number 6
> This is stderr message number 7
> This is stderr message number 8
> This is stderr message number 9
> This is stderr message number 10
> ===== Task Information Footer =====
> End time: Fri Apr 04 19:31:36 PDT 2008
> Exit code: 0
> Input records: 10000
> Input bytes: 1898380 bytes
> Output records: 4
> Output bytes: 219446 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:22 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:22 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:28 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:28 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:32 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:33 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> ===== Task Information Header =====
> Command: ./cplusplus/PigLoggingTest 10 t
> Start time: Fri Apr 04 10:11:37 PDT 2008
> Input-split file: null
> Input-split start-offset: -1
> Input-split length: -1
> ===== * * * =====
> ===== Task Information Footer =====
> End time: Fri Apr 04 10:11:37 PDT 2008
> Exit code: -127
> Input records: 747
> Input bytes: 141796 bytes
> Output records: 0
> Output bytes: 0 bytes (stdout using org.apache.pig.builtin.BinaryStorage)
> ===== * * * =====
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.