[jira] [Commented] (BEAM-8345) Add missing validations to run_rc_validation.sh

["Valentyn Tymofieiev (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-8345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944894#comment-16944894 ] 

Valentyn Tymofieiev commented on BEAM-8345:
-------------------------------------------

I think we are missing Flink/Spark runner validations in automated scripts. When interacting with Flink/Spark, we can run Beam SDK harness in a container or local process. Container execution can also help validate  the  container images we release. 

I manually tried validating Flink/Spark using commands at the end of this message. However that sequence of commands may not be very robust since the output of wordcount is available only inside a container (https://issues.apache.org/jira/browse/BEAM-7857), and the container seems to be garbage-collected shortly after execution. Also, we need to pay attention to which version of Flink we are using. I tried 1.5, but we may need to test against all versions we support.   

Related issue that we may need to address: https://issues.apache.org/jira/browse/BEAM-7858.

{noformat}
git checkout tags/v2.16.0-RC1

./gradlew :runners:spark:job-server:runShadow   or ./gradlew :runners:flink:1.5:job-server:runShadow

In a separate terminal: 

docker pull apachebeam/python3.5_sdk:2.16.0_rc1
docker tag apachebeam/python3.5_sdk:2.16.0_rc1 apachebeam/python3.5_sdk:2.16.0

./gradlew :sdks:python:test-suites:portable:py35:portableWordCountBatch  -PjobEndpoint=localhost:8099 -PenvirionmentType=DOCKER 

As soon as that is done:
docker ps -a  | grep apachebeam
Note the container ID of a running container. It stays around only for a minute or so, and gets garbage-collected...
docker exec 866fb8932207 /bin/bash -c 'cat /tmp/py-wordcount*'

<wordcount output is printed>
{noformat}

> Add missing validations to run_rc_validation.sh
> -----------------------------------------------
>
>                 Key: BEAM-8345
>                 URL: https://issues.apache.org/jira/browse/BEAM-8345
>             Project: Beam
>          Issue Type: Sub-task
>          Components: testing
>            Reporter: Mark Liu
>            Assignee: Mark Liu
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)