You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2021/08/11 11:23:00 UTC
[jira] [Resolved] (PIG-5371) Hdfs bytes written assertions fail in
TestPigRunner
[ https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor resolved PIG-5371.
-------------------------------
Resolution: Cannot Reproduce
> Hdfs bytes written assertions fail in TestPigRunner
> ---------------------------------------------------
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns the byte count not only for the result of pig store operator, but it includes the size of the jar files as well. The problem is this could change very easily, so in my opinion the best would be to remove these assertions from TestPigRunner as this is just causing intermittent and/or persistent failures.
> The test class is for basic testing of PigRunner, and this is achieved well enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns): 57162859
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)