You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Laszlo Gaal (Jira)" <ji...@apache.org> on 2020/01/22 20:58:00 UTC
[jira] [Commented] (IMPALA-8115) some jenkins workers slow to do
real work due to dpkg lock conflicts
[ https://issues.apache.org/jira/browse/IMPALA-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021495#comment-17021495 ]
Laszlo Gaal commented on IMPALA-8115:
-------------------------------------
I have reconfigured several jobs to use labels with names ending in "configured".
These labels run on prebaked AMIs instead of the stock Ubuntu AMIs published by the Ubuntu project.
The prebaked AMIs hav openjdk-8, git and (in some of the cases) Docker preinstalled; unattended upgrades
have also been disabled.
This should make job startup suffer much less latency.
Other jobs (e.g. the Python 2.6 incompatibility checker) have the base AMI updated to a much more recent Ubuntu AMI, making the autoupdate process faster (less delta between the current state and the state recorded in the published AMI).
Unfortunately both solutions need to be reapplied periodically to keep the images and labels up to date.
> some jenkins workers slow to do real work due to dpkg lock conflicts
> --------------------------------------------------------------------
>
> Key: IMPALA-8115
> URL: https://issues.apache.org/jira/browse/IMPALA-8115
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Michael Brown
> Assignee: Laszlo Gaal
> Priority: Major
>
> A Jenkins worker for label {{ubuntu-16.04}} took about 15 minutes to start doing real work. I noticed that it was retrying {{apt-get update}}:
> {noformat}
> ++ sudo apt-get --yes install openjdk-8-jdk
> E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
> E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
> ++ date
> Thu Jan 24 23:37:33 UTC 2019
> ++ sudo apt-get update
> ++ sleep 10
> ++ sudo apt-get --yes install openjdk-8-jdk
> [etc]
> {noformat}
> I ssh'd into a host and saw that, yes, something else was holding onto the dpkg log (confirmed with lsof and not pasted here. dpkg process PID 11459 was the culprit)
> {noformat}
> root 1750 0.0 0.0 4508 1664 ? Ss 23:21 0:00 /bin/sh /usr/lib/apt/apt.systemd.daily
> root 1804 12.3 0.1 141076 80452 ? S 23:22 1:24 \_ /usr/bin/python3 /usr/bin/unattended-upgrade
> root 3263 0.0 0.1 140960 72896 ? S 23:23 0:00 \_ /usr/bin/python3 /usr/bin/unattended-upgrade
> root 11459 0.6 0.0 45920 25184 pts/1 Ss+ 23:24 0:03 \_ /usr/bin/dpkg --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/tzdata_2018i-0ubuntu0.16.04_all.deb /var/cache/apt/archives/distro-info-data_0.28ubuntu0.9_all.deb /var/cache/apt/archives/file_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libmagic1_1%3a5.25-2ubuntu1.1_amd64.deb /var/cache/apt/archives/libisc-export160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns-export162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/isc-dhcp-client_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/isc-dhcp-common_4.3.3-5ubuntu12.9_amd64.deb /var/cache/apt/archives/libidn11_1.32-3ubuntu1.2_amd64.deb /var/cache/apt/archives/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libtasn1-6_4.7-3ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libapparmor-perl_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/apparmor_2.10.95-0ubuntu2.10_amd64.deb /var/cache/apt/archives/curl_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/libgssapi-krb5-2_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5-3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libkrb5support0_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libk5crypto3_1.13.2+dfsg-5ubuntu2.1_amd64.deb /var/cache/apt/archives/libcurl3-gnutls_7.47.0-1ubuntu2.11_amd64.deb /var/cache/apt/archives/apt-transport-https_1.2.29ubuntu0.1_amd64.deb /var/cache/apt/archives/libicu55_55.1-7ubuntu0.4_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/bind9-host_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/dnsutils_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisc160_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libdns162_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccc140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libisccfg140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/liblwres141_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/libbind9-140_1%3a9.10.3.dfsg.P4-8ubuntu1.11_amd64.deb /var/cache/apt/archives/openssl_1.0.2g-1ubuntu4.14_amd64.deb /var/cache/apt/archives/ca-certificates_20170717~16.04.1_all.deb /var/cache/apt/archives/libasprintf0v5_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/gettext-base_0.19.7-2ubuntu3.1_amd64.deb /var/cache/apt/archives/krb5-locales_1.13.2+dfsg-5ubuntu2.1_all.deb /var/cache/apt/archives/libelf1_0.165-3ubuntu1.1_amd64.deb /var/cache/apt/archives/libglib2.0-data_2.48.2-0ubuntu4.1_all.deb /var/cache/apt/archives/libnuma1_2.0.11-1ubuntu1.1_amd64.deb /var/cache/apt/archives/libpolkit-gobject-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libx11-data_2%3a1.6.3-1ubuntu2.1_all.deb /var/cache/apt/archives/libx11-6_2%3a1.6.3-1ubuntu2.1_amd64.deb /var/cache/apt/archives/openssh-sftp-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-server_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/openssh-client_1%3a7.2p2-4ubuntu2.6_amd64.deb /var/cache/apt/archives/rsync_3.1.1-3ubuntu1.2_amd64.deb /var/cache/apt/archives/tcpdump_4.9.2-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/wget_1.17.1-1ubuntu1.4_amd64.deb /var/cache/apt/archives/python3-problem-report_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/python3-apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/apport_2.20.1-0ubuntu2.18_all.deb /var/cache/apt/archives/dns-root-data_2018013001~16.04.1_all.deb /var/cache/apt/archives/dnsmasq-base_2.75-1ubuntu0.16.04.5_amd64.deb /var/cache/apt/archives/git-man_1%3a2.7.4-0ubuntu1.6_all.deb /var/cache/apt/archives/git_1%3a2.7.4-0ubuntu1.6_amd64.deb /var/cache/apt/archives/libmspack0_0.5-1ubuntu0.16.04.3_amd64.deb /var/cache/apt/archives/libpolkit-agent-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/libpolkit-backend-1-0_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/linux-image-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-image-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/linux-aws-headers-4.4.0-1074_4.4.0-1074.84_all.deb /var/cache/apt/archives/linux-headers-4.4.0-1074-aws_4.4.0-1074.84_amd64.deb /var/cache/apt/archives/linux-headers-aws_4.4.0.1074.76_amd64.deb /var/cache/apt/archives/policykit-1_0.105-14.1ubuntu0.4_amd64.deb /var/cache/apt/archives/python3-cryptography_1.2.3-1ubuntu0.2_amd64.deb /var/cache/apt/archives/python3-jwt_1.3.0-1ubuntu0.1_all.deb /var/cache/apt/archives/python3-requests_2.9.1-3ubuntu0.1_all.deb /var/cache/apt/archives/unattended-upgrades_0.90ubuntu0.10_all.deb
> root 12555 0.0 0.0 4500 740 pts/1 S+ 23:24 0:00 \_ /bin/sh /var/lib/dpkg/info/unattended-upgrades.prerm upgrade 0.90ubuntu0.10
> root 12556 0.0 0.0 4500 1736 pts/1 S+ 23:24 0:00 \_ /bin/sh /usr/sbin/invoke-rc.d unattended-upgrades stop
> root 12594 0.0 0.0 24884 1388 pts/1 S+ 23:24 0:00 \_ systemctl stop unattended-upgrades.service
> {noformat}
> After about 15 minutes, the retry to install openjdk-8-jdk succeeded.
> I'm not sure how to make this better except rebuild AMIs with updated packaging. Over time, the system's automatic apt-get update / upgrades are going to take longer and longer and our own stuff is going to block for longer and longer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org