[jira] [Work logged] (BEAM-2871) Add examples of running external libraries on workers

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-2871?focusedWorklogId=86587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-86587 ]

ASF GitHub Bot logged work on BEAM-2871:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 02/Apr/18 14:33
            Start Date: 02/Apr/18 14:33
    Worklog Time Spent: 10m 
      Work Description: rezarokni commented on a change in pull request #3961: [BEAM-2871] Add advanced examples of running external libraries on workers
URL: https://github.com/apache/beam/pull/3961#discussion_r178555698
 
 

 ##########
 File path: examples/java8/src/main/java/org/apache/beam/examples/advanced/subprocess/utils/CallingSubProcessUtils.java
 ##########
 @@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.advanced.subprocess.utils;
+
+import com.google.common.collect.Sets;
+
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+
+import org.apache.beam.examples.advanced.subprocess.configuration.SubProcessConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class for dealing with concurrency and binary file copies to the worker.
+ */
+public class CallingSubProcessUtils {
+
+  // Prevent Instantiation
+  private CallingSubProcessUtils() {};
+
+  static final Logger LOG = LoggerFactory.getLogger(CallingSubProcessUtils.class);
+
+  static boolean initCompleted = false;
+
+  // Allow multiple subclasses to create files, but only one thread per subclass can add the file to
+  // the worker
+  private static final Set<String> downloadedFiles = Sets.<String>newConcurrentHashSet();
+
+  // Limit the number of threads able to do work
+  private static Semaphore semaphore;
+
+  public static void setUp(SubProcessConfiguration configuration, String binaryName)
+      throws Exception {
+
+    if (!initCompleted) {
+
+      if (semaphore == null) {
+        initSemaphore(configuration.getConcurrency());
+      }
+
+      synchronized (downloadedFiles) {
+        if (!downloadedFiles.contains(binaryName)) {
+          // Create Directories if needed
+          FileUtils.createDirectoriesOnWorker(configuration);
+          LOG.info("Calling filesetup to move Executables to worker.");
+          ExecutableFile executableFile = new ExecutableFile(configuration, binaryName);
+          FileUtils.copyFileFromGCSToWorker(executableFile);
+          downloadedFiles.add(binaryName);
+        }
+      }
+      initCompleted = true;
+    }
+  }
+
+  // If you have two ExternalLibraryDoFns in your graph with different concurrency values, this will
 
 Review comment:
   OK, I will add a MAP which will have the DoFn's name stored against it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 86587)
            Time Spent: 10m
    Remaining Estimate: 0h

> Add examples of running external libraries on workers
> -----------------------------------------------------
>
>                 Key: BEAM-2871
>                 URL: https://issues.apache.org/jira/browse/BEAM-2871
>             Project: Beam
>          Issue Type: New Feature
>          Components: examples-java
>    Affects Versions: 2.1.0
>            Reporter: Reza ardeshir rokni
>            Assignee: Reuven Lax
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Provide example for how to run external libraries such as c++ code within a DoFn. 
> More details of this pattern can be viewed in the blog:
> https://cloud.google.com/blog/big-data/2017/07/running-external-libraries-with-cloud-dataflow-for-grid-computing-workloads



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)