You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vihang Karajgaonkar <vi...@cloudera.com> on 2017/03/03 18:03:39 UTC
Pre-commit job failing - No disk space
Hi Sergio,
Looks like the pre-commit jobs are failing due to disk space issues. Can
you please help or let me know how can I fix it?
Thanks,
Vihang
Re: Pre-commit job failing - No disk space
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
I think the issue is coming from TestDbTxnManager test. This test generates
a huge hive.log file (of the order of 25-30 GB). Created HIVE-16109 to fix
this. We should cleanup older logs manually to free the disk space
meanwhile so that pre-commit job is unblocked again.
On Fri, Mar 3, 2017 at 12:29 PM, Sergio Pena <se...@cloudera.com>
wrote:
> Hey Vihang,
>
> Seems there is a UT test causing all this noise. See this:
>
> 30G ./PreCommit-HIVE-Build-3902/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3900/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3899/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3898/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3897/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3896/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3893/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3892/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3891/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3890/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3888/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3887/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 30G ./PreCommit-HIVE-Build-3877/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 29G ./PreCommit-HIVE-Build-3903/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 29G ./PreCommit-HIVE-Build-3894/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 26G ./PreCommit-HIVE-Build-3904/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 26G ./PreCommit-HIVE-Build-3876/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 24G ./PreCommit-HIVE-Build-3895/failed/272_UTBatch_ql_10_
> tests/logs/hive.log
> 19G ./PreCommit-HIVE-Build-3918/failed/272_UTBatch_ql_10_
> tests/logs/.hive.log.9uk9Av
> 5.1G ./PreCommit-HIVE-Build-3913/failed/14-TestCliDriver-
> authorization_create_temp_table.q-skewjoinopt16.q-drop_
> partitions_filter3.q-and-27-more/logs/hive.log
> 4.8G ./PreCommit-HIVE-Build-3913/failed/20-TestCliDriver-
> rename_column.q-union5.q-insert_into1.q-and-27-more/logs/hive.log
> 4.7G ./PreCommit-HIVE-Build-3913/failed/19-TestCliDriver-cp_mj_
> rc.q-order.q-udf_bitwise_shiftleft.q-and-27-more/logs/hive.log
> 4.0G ./PreCommit-HIVE-Build-3918/succeeded/185_UTBatch_service_
> 8_tests/logs/hive.log
> 4.0G ./PreCommit-HIVE-Build-3917/succeeded/185_UTBatch_service_
> 8_tests/logs/hive.log
>
> You can see all the logs from this link:
> http://104.198.109.242/logs/
>
>
>
> On Fri, Mar 3, 2017 at 2:21 PM, Sergio Pena <se...@cloudera.com>
> wrote:
>
>> Hey Vihang,
>>
>> The ptest master shows how the jobs has increased the log size, could you
>> investigate what causes this increase?
>>
>> 44G PreCommit-HIVE-Build-3902
>> 44G PreCommit-HIVE-Build-3900
>> 44G PreCommit-HIVE-Build-3899
>> 44G PreCommit-HIVE-Build-3898
>> 44G PreCommit-HIVE-Build-3897
>> 44G PreCommit-HIVE-Build-3896
>> 44G PreCommit-HIVE-Build-3893
>> 44G PreCommit-HIVE-Build-3892
>> 44G PreCommit-HIVE-Build-3891
>> 44G PreCommit-HIVE-Build-3890
>> 44G PreCommit-HIVE-Build-3888
>> 44G PreCommit-HIVE-Build-3877
>> 43G PreCommit-HIVE-Build-3903
>> 43G PreCommit-HIVE-Build-3894
>> 41G PreCommit-HIVE-Build-3887
>> 40G PreCommit-HIVE-Build-3904
>> 40G PreCommit-HIVE-Build-3876
>> 38G PreCommit-HIVE-Build-3895
>> 16G PreCommit-HIVE-Build-3918
>> 15G PreCommit-HIVE-Build-3917
>> 15G PreCommit-HIVE-Build-3916
>> 15G PreCommit-HIVE-Build-3915
>> 15G PreCommit-HIVE-Build-3913
>> 15G PreCommit-HIVE-Build-3912
>> 15G PreCommit-HIVE-Build-3911
>> 15G PreCommit-HIVE-Build-3910
>> 15G PreCommit-HIVE-Build-3909
>> 15G PreCommit-HIVE-Build-3905
>> 14G PreCommit-HIVE-Build-3914
>> 14G PreCommit-HIVE-Build-3878
>> 212K PreCommit-HIVE-Build-3889
>> 212K PreCommit-HIVE-Build-3880
>> 188K PreCommit-HIVE-Build-3901
>> 156K PreCommit-HIVE-Build-3885
>> 148K PreCommit-HIVE-Build-3884
>> 96K PreCommit-HIVE-Build-3882
>> 88K PreCommit-HIVE-Build-3886
>> 84K PreCommit-HIVE-Build-3881
>> 80K PreCommit-HIVE-Build-3883
>> 60K PreCommit-HIVE-Build-3879
>> 56K PreCommit-HIVE-Build-3908
>>
>> - Sergio
>>
>> On Fri, Mar 3, 2017 at 12:03 PM, Vihang Karajgaonkar <vihang@cloudera.com
>> > wrote:
>>
>>> Hi Sergio,
>>>
>>> Looks like the pre-commit jobs are failing due to disk space issues. Can
>>> you please help or let me know how can I fix it?
>>>
>>> Thanks,
>>> Vihang
>>>
>>
>>
>
Re: Pre-commit job failing - No disk space
Posted by Sergio Pena <se...@cloudera.com>.
Hey Vihang,
Seems there is a UT test causing all this noise. See this:
30G
./PreCommit-HIVE-Build-3902/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3900/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3899/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3898/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3897/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3896/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3893/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3892/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3891/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3890/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3888/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3887/failed/272_UTBatch_ql_10_tests/logs/hive.log
30G
./PreCommit-HIVE-Build-3877/failed/272_UTBatch_ql_10_tests/logs/hive.log
29G
./PreCommit-HIVE-Build-3903/failed/272_UTBatch_ql_10_tests/logs/hive.log
29G
./PreCommit-HIVE-Build-3894/failed/272_UTBatch_ql_10_tests/logs/hive.log
26G
./PreCommit-HIVE-Build-3904/failed/272_UTBatch_ql_10_tests/logs/hive.log
26G
./PreCommit-HIVE-Build-3876/failed/272_UTBatch_ql_10_tests/logs/hive.log
24G
./PreCommit-HIVE-Build-3895/failed/272_UTBatch_ql_10_tests/logs/hive.log
19G
./PreCommit-HIVE-Build-3918/failed/272_UTBatch_ql_10_tests/logs/.hive.log.9uk9Av
5.1G
./PreCommit-HIVE-Build-3913/failed/14-TestCliDriver-authorization_create_temp_table.q-skewjoinopt16.q-drop_partitions_filter3.q-and-27-more/logs/hive.log
4.8G
./PreCommit-HIVE-Build-3913/failed/20-TestCliDriver-rename_column.q-union5.q-insert_into1.q-and-27-more/logs/hive.log
4.7G
./PreCommit-HIVE-Build-3913/failed/19-TestCliDriver-cp_mj_rc.q-order.q-udf_bitwise_shiftleft.q-and-27-more/logs/hive.log
4.0G
./PreCommit-HIVE-Build-3918/succeeded/185_UTBatch_service_8_tests/logs/hive.log
4.0G
./PreCommit-HIVE-Build-3917/succeeded/185_UTBatch_service_8_tests/logs/hive.log
You can see all the logs from this link:
http://104.198.109.242/logs/
On Fri, Mar 3, 2017 at 2:21 PM, Sergio Pena <se...@cloudera.com>
wrote:
> Hey Vihang,
>
> The ptest master shows how the jobs has increased the log size, could you
> investigate what causes this increase?
>
> 44G PreCommit-HIVE-Build-3902
> 44G PreCommit-HIVE-Build-3900
> 44G PreCommit-HIVE-Build-3899
> 44G PreCommit-HIVE-Build-3898
> 44G PreCommit-HIVE-Build-3897
> 44G PreCommit-HIVE-Build-3896
> 44G PreCommit-HIVE-Build-3893
> 44G PreCommit-HIVE-Build-3892
> 44G PreCommit-HIVE-Build-3891
> 44G PreCommit-HIVE-Build-3890
> 44G PreCommit-HIVE-Build-3888
> 44G PreCommit-HIVE-Build-3877
> 43G PreCommit-HIVE-Build-3903
> 43G PreCommit-HIVE-Build-3894
> 41G PreCommit-HIVE-Build-3887
> 40G PreCommit-HIVE-Build-3904
> 40G PreCommit-HIVE-Build-3876
> 38G PreCommit-HIVE-Build-3895
> 16G PreCommit-HIVE-Build-3918
> 15G PreCommit-HIVE-Build-3917
> 15G PreCommit-HIVE-Build-3916
> 15G PreCommit-HIVE-Build-3915
> 15G PreCommit-HIVE-Build-3913
> 15G PreCommit-HIVE-Build-3912
> 15G PreCommit-HIVE-Build-3911
> 15G PreCommit-HIVE-Build-3910
> 15G PreCommit-HIVE-Build-3909
> 15G PreCommit-HIVE-Build-3905
> 14G PreCommit-HIVE-Build-3914
> 14G PreCommit-HIVE-Build-3878
> 212K PreCommit-HIVE-Build-3889
> 212K PreCommit-HIVE-Build-3880
> 188K PreCommit-HIVE-Build-3901
> 156K PreCommit-HIVE-Build-3885
> 148K PreCommit-HIVE-Build-3884
> 96K PreCommit-HIVE-Build-3882
> 88K PreCommit-HIVE-Build-3886
> 84K PreCommit-HIVE-Build-3881
> 80K PreCommit-HIVE-Build-3883
> 60K PreCommit-HIVE-Build-3879
> 56K PreCommit-HIVE-Build-3908
>
> - Sergio
>
> On Fri, Mar 3, 2017 at 12:03 PM, Vihang Karajgaonkar <vi...@cloudera.com>
> wrote:
>
>> Hi Sergio,
>>
>> Looks like the pre-commit jobs are failing due to disk space issues. Can
>> you please help or let me know how can I fix it?
>>
>> Thanks,
>> Vihang
>>
>
>
Re: Pre-commit job failing - No disk space
Posted by Sergio Pena <se...@cloudera.com>.
Hey Vihang,
The ptest master shows how the jobs has increased the log size, could you
investigate what causes this increase?
44G PreCommit-HIVE-Build-3902
44G PreCommit-HIVE-Build-3900
44G PreCommit-HIVE-Build-3899
44G PreCommit-HIVE-Build-3898
44G PreCommit-HIVE-Build-3897
44G PreCommit-HIVE-Build-3896
44G PreCommit-HIVE-Build-3893
44G PreCommit-HIVE-Build-3892
44G PreCommit-HIVE-Build-3891
44G PreCommit-HIVE-Build-3890
44G PreCommit-HIVE-Build-3888
44G PreCommit-HIVE-Build-3877
43G PreCommit-HIVE-Build-3903
43G PreCommit-HIVE-Build-3894
41G PreCommit-HIVE-Build-3887
40G PreCommit-HIVE-Build-3904
40G PreCommit-HIVE-Build-3876
38G PreCommit-HIVE-Build-3895
16G PreCommit-HIVE-Build-3918
15G PreCommit-HIVE-Build-3917
15G PreCommit-HIVE-Build-3916
15G PreCommit-HIVE-Build-3915
15G PreCommit-HIVE-Build-3913
15G PreCommit-HIVE-Build-3912
15G PreCommit-HIVE-Build-3911
15G PreCommit-HIVE-Build-3910
15G PreCommit-HIVE-Build-3909
15G PreCommit-HIVE-Build-3905
14G PreCommit-HIVE-Build-3914
14G PreCommit-HIVE-Build-3878
212K PreCommit-HIVE-Build-3889
212K PreCommit-HIVE-Build-3880
188K PreCommit-HIVE-Build-3901
156K PreCommit-HIVE-Build-3885
148K PreCommit-HIVE-Build-3884
96K PreCommit-HIVE-Build-3882
88K PreCommit-HIVE-Build-3886
84K PreCommit-HIVE-Build-3881
80K PreCommit-HIVE-Build-3883
60K PreCommit-HIVE-Build-3879
56K PreCommit-HIVE-Build-3908
- Sergio
On Fri, Mar 3, 2017 at 12:03 PM, Vihang Karajgaonkar <vi...@cloudera.com>
wrote:
> Hi Sergio,
>
> Looks like the pre-commit jobs are failing due to disk space issues. Can
> you please help or let me know how can I fix it?
>
> Thanks,
> Vihang
>