[jira] [Work logged] (BEAM-6666) subprocess.Popen hangs after use of gRPC channel

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-6666?focusedWorklogId=206164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-206164 ]

ASF GitHub Bot logged work on BEAM-6666:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Mar/19 00:59
            Start Date: 01/Mar/19 00:59
    Worklog Time Spent: 10m 
      Work Description: ihji commented on pull request #7845: [BEAM-6666] subprocess.Popen hangs after use of gRPC channel
URL: https://github.com/apache/beam/pull/7845#discussion_r261447318
 
 

 ##########
 File path: sdks/python/build.gradle
 ##########
 @@ -456,10 +453,39 @@ project.task('crossLanguagePythonJava') {
   dependsOn 'setupVirtualenv'
   dependsOn ':beam-sdks-java-container:docker'
   dependsOn ':beam-runners-core-construction-java:buildTestExpansionServiceJar'
+
   doLast {
+    def testServiceExpansionJar = project(":beam-runners-core-construction-java:").buildTestExpansionServiceJar.archivePath
+    exec {
+      executable 'sh'
+      args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.transforms.external_test --expansion_service_jar=${testServiceExpansionJar}"
+    }
+  }
+}
+
+project.task('crossLanguagePortableWordCount') {
+  dependsOn 'setupVirtualenv'
+  dependsOn ':beam-runners-flink_2.11-job-server-container:docker'
+  dependsOn ':beam-sdks-python-container:docker'
+  dependsOn ':beam-sdks-java-container:docker'
+  dependsOn ':beam-runners-core-construction-java:buildTestExpansionServiceJar'
+
+  doLast {
+    def testServiceExpansionJar = project(":beam-runners-core-construction-java:").buildTestExpansionServiceJar.archivePath
 
 Review comment:
   Thanks for your comment. IMHO, `core-construction-java` is the better place for `ExpansionService` since `ExpansionService` mainly works during the pipeline construction phase. And, because `java-fn-execution` depends on `core-construction-java`, we can still bundle `ExpansionService` with `JobServer` without moving it. When we put `ExpansionService` in `java-fn-execution`, it will block us from creating a class in `core-construction-java` which uses `ExpansionService` class (e.g., external transform for Java SDK and its test).
   
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 206164)
    Time Spent: 2h  (was: 1h 50m)

> subprocess.Popen hangs after use of gRPC channel
> ------------------------------------------------
>
>                 Key: BEAM-6666
>                 URL: https://issues.apache.org/jira/browse/BEAM-6666
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Heejong Lee
>            Assignee: Heejong Lee
>            Priority: Major
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> subprocess.Popen randomly hangs after use of gRPC channel. This makes cross-language wordcount test fail because the test uses Popen to launch Dockerized Flink job server in `PortableRunner.run_pipeline` after use of gRPC channel for the expansion service in `ExternalTransform.expand`. Few symptoms are listed below:
>  * Hanging at `docker_path = check_output(['which', 'docker']).strip()`
>  * Hanging at `self.docker_process = Popen(cmd)`
>  * Crashing with `assertion failed: pthread_mutex_lock(mu) == 0` message
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)