You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Laszlo Bodor (JIRA)" <ji...@apache.org> on 2018/12/15 14:06:00 UTC
[jira] [Comment Edited] (PIG-5371) Hdfs bytes written assertions
fail in TestPigRunner
[ https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722171#comment-16722171 ]
Laszlo Bodor edited comment on PIG-5371 at 12/15/18 2:05 PM:
-------------------------------------------------------------
Thanks [~szita] ! I've two notes on this:
1. -Dtest doesn't filter test cases (all tests will run), -Dtestcase should be used instead
2. 'ant clean jar' is ok, built successfully (I'm on apache trunk), but then 'ant test -Dtestcase=TestPigRunner' hangs, 10 minutes, and counting (i haven't manage to debug it yet
{code:java}
...
test-core:
[delete] Deleting directory /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /tmp/pig_junit_tmp667875539
[echo] Tests in /home/abstractdog/apache/pig/test/excluded-tests-mr will be excluded
[junit] Running org.apache.pig.test.TestPigRunner
{code}
could you please confirm that you can successfully run (and finish) TestPigRunner?
was (Author: abstractdog):
Thanks [~szita] ! I've wo notes on this:
1. -Dtest doesn't filter test cases (all tests will run), -Dtestcase should be used instead
2. 'ant clean jar' is ok, built successfully (I'm on apache trunk), but then 'ant test -Dtestcase=TestPigRunner' hangs, 10 minutes, and counting (i haven't manage to debug it yet
{code}
...
test-core:
[delete] Deleting directory /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /tmp/pig_junit_tmp667875539
[echo] Tests in /home/abstractdog/apache/pig/test/excluded-tests-mr will be excluded
[junit] Running org.apache.pig.test.TestPigRunner
{code}
could you please confirm that you can successfully run (and finish) TestPigRunner?
> Hdfs bytes written assertions fail in TestPigRunner
> ---------------------------------------------------
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
> Reporter: Laszlo Bodor
> Assignee: Laszlo Bodor
> Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns the byte count not only for the result of pig store operator, but it includes the size of the jar files as well. The problem is this could change very easily, so in my opinion the best would be to remove these assertions from TestPigRunner as this is just causing intermittent and/or persistent failures.
> The test class is for basic testing of PigRunner, and this is achieved well enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns): 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)