You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Laszlo Gaal (JIRA)" <ji...@apache.org> on 2018/11/30 14:51:00 UTC
[jira] [Updated] (IMPALA-7913) test-with-docker can sometimes fail
with a ccache fatal error when using CentOS 6 within the containers
[ https://issues.apache.org/jira/browse/IMPALA-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Laszlo Gaal updated IMPALA-7913:
--------------------------------
Description:
When using docker/test-with-docker.py, Impala builds only the core Impala binaries in the "build" phase/container. Test and benchmark executables are built in later phases in the various test containers. These latter builds are executed in parallel in the context of concurrently running containers. These containers separate the running processes, but they share a common ccache directory for efficiency's sake; this directory is managed by the ccache processes running inside the containers.
Recently I have seen multiple ccache failures in this phase, when the containers are hosting centos6 (test-with-docker.py launched with --base-image=centos:6). The failures usually look like this:
{code:java}
2018-11-27 04:59:52.826775 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o
2018-11-27 04:59:52.933599 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_types.cpp.o
2018-11-27 04:59:53.041421 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService.cpp.o
2018-11-27 04:59:53.169103 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_constants.cpp.o
2018-11-27 04:59:53.544171 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_types.cpp.o
2018-11-27 04:59:53.576886 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreSubscriber.cpp.o
2018-11-27 04:59:53.946981 ccache: FATAL: Failed to open /home/impdev/.ccache/tmp/tmp.cpp_stderr.i-20181127-033918.7545: No such file or directory
2018-11-27 04:59:53.947712 make[3]: *** [be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o] Error 1
{code}
The issue seems to be caused by the cache being shared across containers, and running an older ccache version under CentSO 6: [https://ccache.samba.narkive.com/o4BSOjxG/shared-ccache-directory-between-docker-containers] claims that ccache v3.1 (the one CentOS 6 pulls from EPEL by default) has a problem generating unique filenames for its temporary files.
> test-with-docker can sometimes fail with a ccache fatal error when using CentOS 6 within the containers
> -------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-7913
> URL: https://issues.apache.org/jira/browse/IMPALA-7913
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Environment: test-with-docker
> Reporter: Laszlo Gaal
> Assignee: Laszlo Gaal
> Priority: Major
>
> When using docker/test-with-docker.py, Impala builds only the core Impala binaries in the "build" phase/container. Test and benchmark executables are built in later phases in the various test containers. These latter builds are executed in parallel in the context of concurrently running containers. These containers separate the running processes, but they share a common ccache directory for efficiency's sake; this directory is managed by the ccache processes running inside the containers.
> Recently I have seen multiple ccache failures in this phase, when the containers are hosting centos6 (test-with-docker.py launched with --base-image=centos:6). The failures usually look like this:
> {code:java}
> 2018-11-27 04:59:52.826775 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o
> 2018-11-27 04:59:52.933599 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_types.cpp.o
> 2018-11-27 04:59:53.041421 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService.cpp.o
> 2018-11-27 04:59:53.169103 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_constants.cpp.o
> 2018-11-27 04:59:53.544171 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreService_types.cpp.o
> 2018-11-27 04:59:53.576886 [ 23%] Building CXX object be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/StatestoreSubscriber.cpp.o
> 2018-11-27 04:59:53.946981 ccache: FATAL: Failed to open /home/impdev/.ccache/tmp/tmp.cpp_stderr.i-20181127-033918.7545: No such file or directory
> 2018-11-27 04:59:53.947712 make[3]: *** [be/generated-sources/gen-cpp/CMakeFiles/ImpalaThrift.dir/RuntimeProfile_constants.cpp.o] Error 1
> {code}
> The issue seems to be caused by the cache being shared across containers, and running an older ccache version under CentSO 6: [https://ccache.samba.narkive.com/o4BSOjxG/shared-ccache-directory-between-docker-containers] claims that ccache v3.1 (the one CentOS 6 pulls from EPEL by default) has a problem generating unique filenames for its temporary files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org