[jira] [Commented] (IMPALA-11415) Add run-step-wait-all after loading Kudu data

["Laszlo Gaal (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562833#comment-17562833 ] 

Laszlo Gaal commented on IMPALA-11415:
--------------------------------------

There seems to be a writer-to-reader data race in this scenario on the Python files generated by the Thrift compiler.
The reader is{code}
load-data "functional-query" "core" "kudu/none/none" force{code}
which ends up calling {{bin/load-data.py}}. This file starts with a few {{import}} statements, among them{code}
from tests.beeswax.impala_beeswax import *
{code}
This is the {{import}} statement from which most of the tracebacks seen in the failing builds descended from, the stack traces ending in one of the files in shell/gen-py/... , i.e. in one of the Thrift-generated Python files.
The writer is the step{code}
build-and-copy-hive-udfs{code}
This calls into testdata/bin/copy-udfs-udas.sh, which in lines 52-53 invokes {{make}} for a handful of targets:{code}
  "${MAKE_CMD:-make}" ${IMPALA_MAKE_FLAGS} "-j${IMPALA_BUILD_THREADS:-4}" \
      TestUdas TestUdfs test-udfs-ir udfsample udasample udf-sample-ir uda-sample-ir
{code}
Interestingly, this call to {{make}} doesn't just compile the requested modules, but goes back to a very early phase of the build and restarts the Thrift compilation process. This extra step seems to be unnecessary: the files in shell/gen-py/... were already found to be existing and ready after the first Thrift pass (at the beginnins of a full build cycle), quite a bit before dataload proceeded to this step of loading UDFs and UDAs.
I have not yet followed make's trail to the Thrift build step, so I don't know why it is invoked for a second time. It surely looks redundant.

This also shows that [~rizaon]'s suggested fix is an effective way to work around the race: it will delay rewriting the generated Python files so that it happens only after the Kudu loading steps haev finished using the imported Python Thrift modules.
I am still curious if the second Thrift compilation pass can be eliminiated.

> Add run-step-wait-all after loading Kudu data
> ---------------------------------------------
>
>                 Key: IMPALA-11415
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11415
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Riza Suminto
>            Priority: Major
>              Labels: broken-build
>
> IMPALA-11384 reveals an issue in testdata/bin/create-load-data.sh.
> {code:java}
> if [[ $SKIP_METADATA_LOAD -eq 1 ]]; then
>   # Tests depend on the kudu data being clean, so load the data from scratch.
>   # This is only necessary if this is not a full dataload, because a full dataload
>   # already loads Kudu functional and TPC-H tables from scratch.
>   run-step-backgroundable "Loading Kudu functional" load-kudu.log \
>         load-data "functional-query" "core" "kudu/none/none" force
>   run-step-backgroundable "Loading Kudu TPCH" load-kudu-tpch.log \
>         load-data "tpch" "core" "kudu/none/none" force
> fi
> run-step-backgroundable "Loading Hive UDFs" build-and-copy-hive-udfs.log \
>     build-and-copy-hive-udfs {code}
> If $SKIP_METADATA_LOAD is true, all three of "Loading Kudu functional", "Loading Kudu TPCH", and "Loading Hive UDFs" will be run in parallel in the background. The later background step seemingly override the thrift generated python code under shell/gen-py/hive_metastore/ and shell/gen-py/beeswaxd/. This in turn cause sporadic python error upon invocation of bin/load-data.py of the two former Kudu background steps. Adding run-step-wait-all after the Kudu data loading seems to fix the issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org