You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/17 17:16:23 UTC

[GitHub] [airflow] pierrejeambrun opened a new pull request #20386: Add support for BeamGoPipelineOperator

pierrejeambrun opened a new pull request #20386:
URL: https://github.com/apache/airflow/pull/20386


   closes: https://github.com/apache/airflow/issues/20283
   
   In this PR:
   - [x]  Upgrade the minimum package requirement to 2.33.0 for apache-beam (first stable for beam go sdk)
   - [x]  Refactor `operators/beam.py` with an abstract `BeamBasePipelineOperator` class to factorize initialization and common code, also fixed mypy hook on ``BeamDataflowMixin``
   - [x] Add `BeamRunGoPipelineOperator` and `BeamHook.start_go_pipeline` (+tests)
   - [x]  Add `utils/go_module.py` to handle initialisation and dependency installation for a module. (+ tests)
   - [x]  Slightly modified `process_util` + tests to be able to handle an extra optional parameter `cwd`. (This way we can move to the module directory to build it)
   - [x]  Write docs
   - [x]  Add dags examples, with system tests
   - [x]  Run pre-commits
   - [ ]  We need to install Go in docker images to be able to run it. Go 1.16 minimum for beam 2.33
   
   ``` bash
   wget https://dl.google.com/go/go1.16.4.linux-amd64.tar.gz
   sudo tar -xvf go1.16.4.linux-amd64.tar.gz   
   sudo mv go /usr/local 
   # in bashrc
   export GOROOT=/usr/local/go
   export PATH=$GOPATH/bin:$GOROOT/bin:$PATH 
   ```
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777569675



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-997892069


   @subkanthi Thank you for pointing. There were 2 definitions of the same `_jobs` attribute, I removed one in my branch and `main` removed the other :face_with_spiral_eyes:, hence the missing attribute.
   
   I just fixed that, and also rebased my branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775210194



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")

Review comment:
       We should not assume that the directory `/tmp/` is a directory for temporary files. Some users have a temporary directory in a different location. See: https://github.com/apache/airflow/issues/19206




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038010070


   Hello @potiuk,
   
   I hope you are doing well :smile: 
   
   The CI has failed on `Tests / Helm Chart; KubernetesExecutor` (1 test didn't pass) but it does not seem related. I'm not familiar with it, should we relaunch the pipeline ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1008423832


   I just rebased on main and updated the PR following your reviews. It is not yet ready to merge, I am having an issue running the go pipeline on dataflow, looks like he is not using the correct docker image even with `WorkerHarnessContainerImage: apache/beam_go_sdk:latest` option 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r778238633



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options

Review comment:
       Fixed, also modified the same mistake in the `dataflow` operator.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] aaltay commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1009524420


   > I just rebased on main and updated the PR following your reviews. It is not yet ready to merge, I am having an issue running the go pipeline on dataflow, looks like he is not using the correct image even with the `WorkerHarnessContainerImage: apache/beam_go_sdk:latest` option
   
   If you cannot figure out, @jrmccluskey could help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777532368



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -96,7 +100,107 @@ def set_current_dataflow_job_id(job_id):
         )
 
 
-class BeamRunPythonPipelineOperator(BaseOperator, BeamDataflowMixin):
+class BeamBasePipelineOperator(BaseOperator, BeamDataflowMixin, ABC):
+    """
+    Abstract base class for Beam Pipeline Operators.
+
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+
+    :type runner: str
+    :param default_pipeline_options: Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.

Review comment:
       Not sure this is relevant indeed, It was like this in the child classes, I have removed it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035967006


   > @potiuk Yes of course, I just rebased the branch. Nice I didn't know we could do that directly from the GitHub interface +1
   
   It's a new feature (added last week) which we turned on immediately after it appeared (that's why I am also spreading the news about it in comments :) ). 
   
   Unfortunately - static checks you will have to fix by pushing new code I am afraid :smiley: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1008423832


   I just rebased on main and updated the PR following your reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] aaltay commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r773601098



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -185,56 +289,30 @@ def __init__(
         dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
         **kwargs,
     ) -> None:
-        super().__init__(**kwargs)
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
 
         self.py_file = py_file
-        self.runner = runner
         self.py_options = py_options or []
-        self.default_pipeline_options = default_pipeline_options or {}
-        self.pipeline_options = pipeline_options or {}
-        self.pipeline_options.setdefault("labels", {}).update(
-            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
-        )
         self.py_interpreter = py_interpreter
         self.py_requirements = py_requirements
         self.py_system_site_packages = py_system_site_packages
-        self.gcp_conn_id = gcp_conn_id
-        self.delegate_to = delegate_to
-        self.beam_hook: Optional[BeamHook] = None
-        self.dataflow_hook: Optional[DataflowHook] = None
-        self.dataflow_job_id: Optional[str] = None
-
-        if dataflow_config is None:
-            self.dataflow_config = DataflowConfiguration()
-        elif isinstance(dataflow_config, dict):
-            self.dataflow_config = DataflowConfiguration(**dataflow_config)
-        else:
-            self.dataflow_config = dataflow_config
-
-        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
-            self.log.warning(
-                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
-            )
 
     def execute(self, context):
         """Execute the Apache Beam Pipeline."""
-        self.beam_hook = BeamHook(runner=self.runner)
-        pipeline_options = self.default_pipeline_options.copy()
-        process_line_callback: Optional[Callable] = None
-        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
-        dataflow_job_name: Optional[str] = None
-
-        if is_dataflow:
-            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
-                pipeline_options=pipeline_options, job_name_variable_key="job_name"
-            )
-
-        pipeline_options.update(self.pipeline_options)
-
-        # Convert argument names from lowerCamelCase to snake case.
-        formatted_pipeline_options = {
-            convert_camel_to_snake(key): pipeline_options[key] for key in pipeline_options
-        }
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       What is job_name_variable_key ?

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -96,7 +100,107 @@ def set_current_dataflow_job_id(job_id):
         )
 
 
-class BeamRunPythonPipelineOperator(BaseOperator, BeamDataflowMixin):
+class BeamBasePipelineOperator(BaseOperator, BeamDataflowMixin, ABC):
+    """
+    Abstract base class for Beam Pipeline Operators.
+
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+
+    :type runner: str
+    :param default_pipeline_options: Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: Optional.
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  Optional.
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: Dataflow configuration, used when runner type is set to DataflowRunner,
+        (optional) defaults to None.
+    :type dataflow_config: DataflowConfiguration
+    """
+
+    def __init__(
+        self,
+        *,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.runner = runner
+        self.default_pipeline_options = default_pipeline_options or {}
+        self.pipeline_options = pipeline_options or {}
+        self.pipeline_options.setdefault("labels", {}).update(
+            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
+        )
+        self.gcp_conn_id = gcp_conn_id
+        self.delegate_to = delegate_to
+        if isinstance(dataflow_config, dict):
+            self.dataflow_config = DataflowConfiguration(**dataflow_config)
+        else:
+            self.dataflow_config = dataflow_config or DataflowConfiguration()
+        self.beam_hook: Optional[BeamHook] = None
+        self.dataflow_hook: Optional[DataflowHook] = None
+        self.dataflow_job_id: Optional[str] = None
+
+        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
+            self.log.warning(
+                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
+            )
+
+    def _init_pipeline_options(
+        self,
+        format_pipeline_options: bool = False,
+        job_name_variable_key: Optional[str] = None,
+    ) -> Tuple[bool, Optional[str], dict, Optional[Callable[[str], None]]]:
+        self.beam_hook = BeamHook(runner=self.runner)
+        pipeline_options = self.default_pipeline_options.copy()
+        process_line_callback: Optional[Callable[[str], None]] = None
+        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
+        dataflow_job_name: Optional[str] = None
+
+        if is_dataflow:
+            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
+                pipeline_options=pipeline_options,
+                job_name_variable_key=job_name_variable_key,
+            )
+
+        pipeline_options.update(self.pipeline_options)
+
+        formatted_pipeline_options = pipeline_options

Review comment:
       Perhaps rename this to something more explicit like format_as_snake case, or something like that.

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       should go also be formatting the options like python?

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -96,7 +100,107 @@ def set_current_dataflow_job_id(job_id):
         )
 
 
-class BeamRunPythonPipelineOperator(BaseOperator, BeamDataflowMixin):
+class BeamBasePipelineOperator(BaseOperator, BeamDataflowMixin, ABC):
+    """
+    Abstract base class for Beam Pipeline Operators.
+
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+
+    :type runner: str
+    :param default_pipeline_options: Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.

Review comment:
       why is labels special in this case?

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.

Review comment:
       Is it possible to avoid these repated doc strings and refer to the base class?

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       should this be inside is dataflow case? what is dataflow job id otherwise?

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+

Review comment:
       remove empty line?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775209805



##########
File path: tests/utils/test_process_utils.py
##########
@@ -97,6 +97,10 @@ def test_reap_process_group(self):
 
 
 class TestExecuteInSubProcess(unittest.TestCase):
+    @pytest.fixture(autouse=True)
+    def _tmp_path(self, tmp_path):
+        self.tmp_path = tmp_path

Review comment:
       Why is this added to all tests in this class? I guess this will only be used in one method so we can do it at method level.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777568524



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       From what I have seen in the [quick start](https://beam.apache.org/get-started/quickstart-go/), it looks like indeed go takes parameters as snake case. This way we can correct if someone enter them with camel case and it will still work.
   
   I think this is nice but I can remove it if necessary.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777571789



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       If the runner is not `dataflow`, this will default to `None`. Indeed I don't make much sense to return a `dataflow_job_id` when we are not in the dataflow case. I wonder if this is for backward compatibility reasons.

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       If the runner is not `dataflow`, this will default to `None`. Indeed It doesn't make much sense to return a `dataflow_job_id` when we are not in the dataflow case. I wonder if this is for backward compatibility reasons.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038027380


   IT's a flaky test. No worries
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk merged pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20386:
URL: https://github.com/apache/airflow/pull/20386


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-996922608


   Just amended for fix static checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r778238633



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options

Review comment:
       Fixed, also modified the same mistake in the `google cloud dataflow` operator




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777540009



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -96,7 +100,107 @@ def set_current_dataflow_job_id(job_id):
         )
 
 
-class BeamRunPythonPipelineOperator(BaseOperator, BeamDataflowMixin):
+class BeamBasePipelineOperator(BaseOperator, BeamDataflowMixin, ABC):
+    """
+    Abstract base class for Beam Pipeline Operators.
+
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+
+    :type runner: str
+    :param default_pipeline_options: Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: Optional.
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  Optional.
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: Dataflow configuration, used when runner type is set to DataflowRunner,
+        (optional) defaults to None.
+    :type dataflow_config: DataflowConfiguration
+    """
+
+    def __init__(
+        self,
+        *,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.runner = runner
+        self.default_pipeline_options = default_pipeline_options or {}
+        self.pipeline_options = pipeline_options or {}
+        self.pipeline_options.setdefault("labels", {}).update(
+            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
+        )
+        self.gcp_conn_id = gcp_conn_id
+        self.delegate_to = delegate_to
+        if isinstance(dataflow_config, dict):
+            self.dataflow_config = DataflowConfiguration(**dataflow_config)
+        else:
+            self.dataflow_config = dataflow_config or DataflowConfiguration()
+        self.beam_hook: Optional[BeamHook] = None
+        self.dataflow_hook: Optional[DataflowHook] = None
+        self.dataflow_job_id: Optional[str] = None
+
+        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
+            self.log.warning(
+                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
+            )
+
+    def _init_pipeline_options(
+        self,
+        format_pipeline_options: bool = False,
+        job_name_variable_key: Optional[str] = None,
+    ) -> Tuple[bool, Optional[str], dict, Optional[Callable[[str], None]]]:
+        self.beam_hook = BeamHook(runner=self.runner)
+        pipeline_options = self.default_pipeline_options.copy()
+        process_line_callback: Optional[Callable[[str], None]] = None
+        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
+        dataflow_job_name: Optional[str] = None
+
+        if is_dataflow:
+            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
+                pipeline_options=pipeline_options,
+                job_name_variable_key=job_name_variable_key,
+            )
+
+        pipeline_options.update(self.pipeline_options)
+
+        formatted_pipeline_options = pipeline_options

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1037129646






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-996922608


   Just amended to fix static checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r778291137



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.

Review comment:
       Yes, I added the `PortableRunner as other possible option`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777552948



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -185,56 +289,30 @@ def __init__(
         dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
         **kwargs,
     ) -> None:
-        super().__init__(**kwargs)
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
 
         self.py_file = py_file
-        self.runner = runner
         self.py_options = py_options or []
-        self.default_pipeline_options = default_pipeline_options or {}
-        self.pipeline_options = pipeline_options or {}
-        self.pipeline_options.setdefault("labels", {}).update(
-            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
-        )
         self.py_interpreter = py_interpreter
         self.py_requirements = py_requirements
         self.py_system_site_packages = py_system_site_packages
-        self.gcp_conn_id = gcp_conn_id
-        self.delegate_to = delegate_to
-        self.beam_hook: Optional[BeamHook] = None
-        self.dataflow_hook: Optional[DataflowHook] = None
-        self.dataflow_job_id: Optional[str] = None
-
-        if dataflow_config is None:
-            self.dataflow_config = DataflowConfiguration()
-        elif isinstance(dataflow_config, dict):
-            self.dataflow_config = DataflowConfiguration(**dataflow_config)
-        else:
-            self.dataflow_config = dataflow_config
-
-        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
-            self.log.warning(
-                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
-            )
 
     def execute(self, context):
         """Execute the Apache Beam Pipeline."""
-        self.beam_hook = BeamHook(runner=self.runner)
-        pipeline_options = self.default_pipeline_options.copy()
-        process_line_callback: Optional[Callable] = None
-        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
-        dataflow_job_name: Optional[str] = None
-
-        if is_dataflow:
-            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
-                pipeline_options=pipeline_options, job_name_variable_key="job_name"
-            )
-
-        pipeline_options.update(self.pipeline_options)
-
-        # Convert argument names from lowerCamelCase to snake case.
-        formatted_pipeline_options = {
-            convert_camel_to_snake(key): pipeline_options[key] for key in pipeline_options
-        }
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       This is an existing logic that allows us to set an alternate key in the pipeline options with the `job_name` value (only for dataflow). This will set the key to our `job_name_variable_key` and the value to the actual `job_name`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777571789



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       If the runner is not `dataflow`, this will default to `None`. Indeed It doesn't make much sense to return a `dataflow_job_id` when we are not in the dataflow case. I wonder if this is for backward compatibility reasons.
   
   I will try to change that behavior

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       If the runner is not `dataflow`, this will default to `None`. Indeed It doesn't make much sense to return a `dataflow_job_id` when we are not in the dataflow case. I wonder if this is for backward compatibility reasons.
   
   I will try to change that behavior
   
   Edit: done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777568524



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       From what I have seen in the [quick start](https://beam.apache.org/get-started/quickstart-go/), it looks like Go also takes snake case parameters. This way we can correct if someone enter them with camel case and it will still work.
   
   I think this is nice but I can remove it if necessary.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775209930



##########
File path: airflow/providers/google/cloud/hooks/gcs.py
##########
@@ -344,7 +344,7 @@ def provide_file(
         self,
         bucket_name: str = PROVIDE_BUCKET,
         object_name: Optional[str] = None,
-        object_url: Optional[str] = None,
+        dir: Optional[str] = None,

Review comment:
       Can you tell a little more about it? `object_url` parameter is used by `_fallback_object_url_to_object_name_and_bucket_name` decorator, so if we want the correct typing then we should add this parameter back.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777564854



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.

Review comment:
       Yes you are right. I took a look at the apache-cassandra and aws-athena providers, looks like we do not specify base class parameters to avoid duplicates.
   
   I removed the duplicates in the base class.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r780827042



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")

Review comment:
       You are right, thank you. Fixed using the `tempfile.TemporaryDirectory`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038010070


   Hello @potiuk,
   
   I hope you are doing well :smile: 
   
   The CI has failed on `Tests / Helm Chart; KubernetesExecutor` (1 test didn't pass) but it does not seem related. I'm not familiar with it, should we relaunch the pipeline ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038029380


   I am doing some more changes in the images, so I might add Go to the image. Can you please add and issue about it @pierrejeambrun and CC: me ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1017958438


   Just tested it, everything is working fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] subkanthi commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
subkanthi commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-997825088


   The dataflow tests seems to be failing, guessing its related to changing apache-beam version.
   
   ` tests/providers/google/cloud/hooks/test_dataflow.py .................... [ 40%]
     .......................................................FFFFFFFFF........ [ 41%]`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775209640



##########
File path: tests/utils/test_process_utils.py
##########
@@ -105,6 +109,20 @@ def test_should_print_all_messages1(self):
 
         assert ["Executing cmd: bash -c 'echo CAT; echo KITTY;'", 'Output:', 'CAT', 'KITTY'] == msgs
 
+    def test_should_print_all_messages_from_cwd(self):

Review comment:
       ```suggestion
       def test_should_print_all_messages_from_cwd(self, tmp_path):
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] aaltay commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-999291369


   I will be OOO soon, and in my absence @jrmccluskey could continue the review. Thank you for adding this operator!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777552948



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -185,56 +289,30 @@ def __init__(
         dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
         **kwargs,
     ) -> None:
-        super().__init__(**kwargs)
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
 
         self.py_file = py_file
-        self.runner = runner
         self.py_options = py_options or []
-        self.default_pipeline_options = default_pipeline_options or {}
-        self.pipeline_options = pipeline_options or {}
-        self.pipeline_options.setdefault("labels", {}).update(
-            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
-        )
         self.py_interpreter = py_interpreter
         self.py_requirements = py_requirements
         self.py_system_site_packages = py_system_site_packages
-        self.gcp_conn_id = gcp_conn_id
-        self.delegate_to = delegate_to
-        self.beam_hook: Optional[BeamHook] = None
-        self.dataflow_hook: Optional[DataflowHook] = None
-        self.dataflow_job_id: Optional[str] = None
-
-        if dataflow_config is None:
-            self.dataflow_config = DataflowConfiguration()
-        elif isinstance(dataflow_config, dict):
-            self.dataflow_config = DataflowConfiguration(**dataflow_config)
-        else:
-            self.dataflow_config = dataflow_config
-
-        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
-            self.log.warning(
-                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
-            )
 
     def execute(self, context):
         """Execute the Apache Beam Pipeline."""
-        self.beam_hook = BeamHook(runner=self.runner)
-        pipeline_options = self.default_pipeline_options.copy()
-        process_line_callback: Optional[Callable] = None
-        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
-        dataflow_job_name: Optional[str] = None
-
-        if is_dataflow:
-            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
-                pipeline_options=pipeline_options, job_name_variable_key="job_name"
-            )
-
-        pipeline_options.update(self.pipeline_options)
-
-        # Convert argument names from lowerCamelCase to snake case.
-        formatted_pipeline_options = {
-            convert_camel_to_snake(key): pipeline_options[key] for key in pipeline_options
-        }
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       This is an existing logic that allows us to set an alternate key in the pipeline options with the `dataflow_job_name` value (only for dataflow). This will set the key to our `job_name_variable_key` and the value to the actual `job_name`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r778238633



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options

Review comment:
       Fixed, also modified the same mistake in the `google/cloud/operators/dataflow` file.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r780826734



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside

Review comment:
       Yes of course, done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r780820054



##########
File path: tests/utils/test_process_utils.py
##########
@@ -105,6 +109,20 @@ def test_should_print_all_messages1(self):
 
         assert ["Executing cmd: bash -c 'echo CAT; echo KITTY;'", 'Output:', 'CAT', 'KITTY'] == msgs
 
+    def test_should_print_all_messages_from_cwd(self):

Review comment:
       `unittest.TestCase` are only compatible with `autouse` fixture. I have changed a little bit the test so I don't need the `self.assertLogs` method. This way I can remove the extends to `unittest.TestCase` and use normal fixtures.
   
   Let me know if this is better.

##########
File path: tests/utils/test_process_utils.py
##########
@@ -97,6 +97,10 @@ def test_reap_process_group(self):
 
 
 class TestExecuteInSubProcess(unittest.TestCase):
+    @pytest.fixture(autouse=True)
+    def _tmp_path(self, tmp_path):
+        self.tmp_path = tmp_path

Review comment:
       Removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1008423832






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038029380


   I am doing some more changes in the images, so I might add Go to the image. Can you please add an issue about it @pierrejeambrun and CC: me ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777552948



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -185,56 +289,30 @@ def __init__(
         dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
         **kwargs,
     ) -> None:
-        super().__init__(**kwargs)
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
 
         self.py_file = py_file
-        self.runner = runner
         self.py_options = py_options or []
-        self.default_pipeline_options = default_pipeline_options or {}
-        self.pipeline_options = pipeline_options or {}
-        self.pipeline_options.setdefault("labels", {}).update(
-            {"airflow-version": "v" + version.replace(".", "-").replace("+", "-")}
-        )
         self.py_interpreter = py_interpreter
         self.py_requirements = py_requirements
         self.py_system_site_packages = py_system_site_packages
-        self.gcp_conn_id = gcp_conn_id
-        self.delegate_to = delegate_to
-        self.beam_hook: Optional[BeamHook] = None
-        self.dataflow_hook: Optional[DataflowHook] = None
-        self.dataflow_job_id: Optional[str] = None
-
-        if dataflow_config is None:
-            self.dataflow_config = DataflowConfiguration()
-        elif isinstance(dataflow_config, dict):
-            self.dataflow_config = DataflowConfiguration(**dataflow_config)
-        else:
-            self.dataflow_config = dataflow_config
-
-        if self.dataflow_config and self.runner.lower() != BeamRunnerType.DataflowRunner.lower():
-            self.log.warning(
-                "dataflow_config is defined but runner is different than DataflowRunner (%s)", self.runner
-            )
 
     def execute(self, context):
         """Execute the Apache Beam Pipeline."""
-        self.beam_hook = BeamHook(runner=self.runner)
-        pipeline_options = self.default_pipeline_options.copy()
-        process_line_callback: Optional[Callable] = None
-        is_dataflow = self.runner.lower() == BeamRunnerType.DataflowRunner.lower()
-        dataflow_job_name: Optional[str] = None
-
-        if is_dataflow:
-            dataflow_job_name, pipeline_options, process_line_callback = self._set_dataflow(
-                pipeline_options=pipeline_options, job_name_variable_key="job_name"
-            )
-
-        pipeline_options.update(self.pipeline_options)
-
-        # Convert argument names from lowerCamelCase to snake case.
-        formatted_pipeline_options = {
-            convert_camel_to_snake(key): pipeline_options[key] for key in pipeline_options
-        }
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")

Review comment:
       This is an existing logic that allows us to set an alternate key in the pipeline options to set the job name (only for dataflow). This will set the key to our `job_name_variable_key` and the value to the actual `job_name`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jrmccluskey commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
jrmccluskey commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r773961221



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.

Review comment:
       The Python Portable Runner (handled as the "universal" runner as far as flags are concerned) is also an option. Same situation as with Spark and Flink, just need to set `--runner` to universal and provide the correct endpoint. 

##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options

Review comment:
       ```suggestion
             If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key=B`` options
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775210554



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside

Review comment:
       Can you use TemporaryDirectory to create a new temporary file and GCSHook.download to download a file to this directory?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1017958438


   Just tested it, normally everything is working fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1013950852


   Thank you @aaltay, I managed to make it work (go pipeline on dataflow), the issue was on my end caused by a wrong configuration.
   
   I have rebased my branch, and tested {python, go} * {direct, dataflow} everything is working as expected.
   
   I have not yet tested java. I will try to do it this week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035527469


   @mik-laj - any more comments on that ? @pierrejeambrun  - can you please rebase to latest main to check if there are no static checks/doc changes (the change is 179 commits behind main). There is the new feature from GitHub that makes it super - easy.
   
   <img width="1252" alt="Screenshot 2022-02-10 at 22 16 04" src="https://user-images.githubusercontent.com/595491/153497773-90fd43b1-e5ce-4c67-acd5-2d15f0a0377d.png">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035619367


   @potiuk Yes of course, I just rebased the branch. Nice I didn't know we could do that directly from the GitHub interface :+1:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777615970



##########
File path: airflow/providers/google/cloud/hooks/gcs.py
##########
@@ -344,7 +344,7 @@ def provide_file(
         self,
         bucket_name: str = PROVIDE_BUCKET,
         object_name: Optional[str] = None,
-        object_url: Optional[str] = None,
+        dir: Optional[str] = None,

Review comment:
       You are right. I missed that the decorator was changing the signature and that this was used for typing.
   
   I reverted this change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-998814773


   @aaltay @ibzib @kennknowles @TheNeuralBit Can I ask for a review? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1008423832


   I just rebased on main and updated the PR following your reviews. It is not yet ready to merge, I am having an issue running the go dataflow pipeline.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1024592494


   @mik-laj The PR is ready if you want to take a look :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r775209432



##########
File path: airflow/utils/go_module.py
##########
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""Utilities initializing and managing Go modules."""

Review comment:
       Airflow providers need to be compatible with Airflow 2.1, so operators cannot depend on new modules in the core. Can you copy this module to the google provider?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-997892069


   @subkanthi Thank you for pointing that. There were 2 definitions of the same `_jobs` attribute, I removed one in my branch and `main` removed the other :face_with_spiral_eyes:, hence the missing attribute.
   
   I just fixed that, and also rebased my branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1035458565


   Hi @eladkal and @potiuk,
   
   It has been a couple of week since last activity on this PR. I would like to follow up, I think it's ready to merge.
   
   Is there anything I can do to help you or changes that need to be made ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1036572628


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk merged pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20386:
URL: https://github.com/apache/airflow/pull/20386


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1037129646


   > Unfortunately - static checks you will have to fix by pushing new code I am afraid :smile:
   
   @potiuk  that new docstring-param hook got me I guess :joy:
   
   I just pushed a fix for that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038029380


   I am doing some more changes in the images, so I might add Go to the image. Can you please add an issue about it @pierrejeambrun and CC: me ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#issuecomment-1038027380






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777564854



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.

Review comment:
       Yes you are right. I took a look at the Cassandra provider, looks like we do not specify base class parameters to avoid duplicates.
   
   I removed the duplicates in the base class.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pierrejeambrun commented on a change in pull request #20386: Add support for BeamGoPipelineOperator

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on a change in pull request #20386:
URL: https://github.com/apache/airflow/pull/20386#discussion_r777571789



##########
File path: airflow/providers/apache/beam/operators/beam.py
##########
@@ -470,3 +529,144 @@ def on_kill(self) -> None:
                 job_id=self.dataflow_job_id,
                 project_id=self.dataflow_config.project_id,
             )
+
+
+class BeamRunGoPipelineOperator(BeamBasePipelineOperator):
+    """
+    Launching Apache Beam pipelines written in Go. Note that both
+    ``default_pipeline_options`` and ``pipeline_options`` will be merged to specify pipeline
+    execution parameter, and ``default_pipeline_options`` is expected to save
+    high-level options, for instances, project and zone information, which
+    apply to all beam operators in the DAG.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:BeamRunGoPipelineOperator`
+
+    .. seealso::
+        For more detail on Apache Beam have a look at the reference:
+        https://beam.apache.org/documentation/
+
+    :param go_file: Reference to the Go Apache Beam pipeline e.g.,
+        /some/local/file/path/to/your/go/pipeline/file.go
+    :type go_file: str
+    :param runner: Runner on which pipeline will be run. By default "DirectRunner" is being used.
+        Other possible options: DataflowRunner, SparkRunner, FlinkRunner.
+        See: :class:`~providers.apache.beam.hooks.beam.BeamRunnerType`
+        See: https://beam.apache.org/documentation/runners/capability-matrix/
+    :type runner: str
+    :param default_pipeline_options: (optional) Map of default pipeline options.
+    :type default_pipeline_options: dict
+    :param pipeline_options: (optional) Map of pipeline options.The key must be a dictionary.
+        The value can contain different types:
+
+        * If the value is None, the single option - ``--key`` (without value) will be added.
+        * If the value is False, this option will be skipped
+        * If the value is True, the single option - ``--key`` (without value) will be added.
+        * If the value is list, the many options will be added for each key.
+          If the value is ``['A', 'B']`` and the key is ``key`` then the ``--key=A --key-B`` options
+          will be left
+        * Other value types will be replaced with the Python textual representation.
+
+        When defining labels (``labels`` option), you can also provide a dictionary.
+    :type pipeline_options: dict
+    :param gcp_conn_id: (optional)
+        The connection ID to use connecting to Google Cloud Storage if python file is on GCS.
+    :type gcp_conn_id: str
+    :param delegate_to:  (optional)
+        The account to impersonate using domain-wide delegation of authority,
+        if any. For this to work, the service account making the request must have
+        domain-wide delegation enabled.
+    :type delegate_to: str
+    :param dataflow_config: (optional) Dataflow configuration, used when runner type is set to DataflowRunner
+    :type dataflow_config: Union[dict, providers.google.cloud.operators.dataflow.DataflowConfiguration]
+    """
+
+    template_fields = [
+        "go_file",
+        "runner",
+        "pipeline_options",
+        "default_pipeline_options",
+        "dataflow_config",
+    ]
+    template_fields_renderers = {'dataflow_config': 'json', 'pipeline_options': 'json'}
+
+    def __init__(
+        self,
+        *,
+        go_file: str,
+        runner: str = "DirectRunner",
+        default_pipeline_options: Optional[dict] = None,
+        pipeline_options: Optional[dict] = None,
+        gcp_conn_id: str = "google_cloud_default",
+        delegate_to: Optional[str] = None,
+        dataflow_config: Optional[Union[DataflowConfiguration, dict]] = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(
+            runner=runner,
+            default_pipeline_options=default_pipeline_options,
+            pipeline_options=pipeline_options,
+            gcp_conn_id=gcp_conn_id,
+            delegate_to=delegate_to,
+            dataflow_config=dataflow_config,
+            **kwargs,
+        )
+
+        self.go_file = go_file
+        self.should_init_go_module = False
+
+    def execute(self, context):
+        """Execute the Apache Beam Pipeline."""
+        (
+            is_dataflow,
+            dataflow_job_name,
+            formatted_pipeline_options,
+            process_line_callback,
+        ) = self._init_pipeline_options(format_pipeline_options=True, job_name_variable_key="job_name")
+
+        with ExitStack() as exit_stack:
+            if self.go_file.lower().startswith("gs://"):
+                gcs_hook = GCSHook(self.gcp_conn_id, self.delegate_to)
+                # Go doesn't allow `go mod tidy` inside root tmp dir, creating a go subdir to put files inside
+                if not os.path.isdir("/tmp/go"):
+                    os.mkdir("/tmp/go")
+                tmp_gcs_file = exit_stack.enter_context(
+                    gcs_hook.provide_file(object_url=self.go_file, dir="/tmp/go")
+                )
+                self.go_file = tmp_gcs_file.name
+                self.should_init_go_module = True
+
+            if is_dataflow:
+                with self.dataflow_hook.provide_authorized_gcloud():
+                    self.beam_hook.start_go_pipeline(
+                        variables=formatted_pipeline_options,
+                        go_file=self.go_file,
+                        process_line_callback=process_line_callback,
+                        should_init_module=self.should_init_go_module,
+                    )
+
+                self.dataflow_hook.wait_for_done(
+                    job_name=dataflow_job_name,
+                    location=self.dataflow_config.location,
+                    job_id=self.dataflow_job_id,
+                    multiple_jobs=False,
+                )
+
+            else:
+                self.beam_hook.start_go_pipeline(
+                    variables=formatted_pipeline_options,
+                    go_file=self.go_file,
+                    process_line_callback=process_line_callback,
+                    should_init_module=self.should_init_go_module,
+                )
+
+        return {"dataflow_job_id": self.dataflow_job_id}

Review comment:
       If the runner is not `dataflow`, this will default to `None`. Indeed It doesn't make much sense to return a `dataflow_job_id` when we are not in the dataflow case. I wonder if this is for backward compatibility reasons.
   
   Not sure this will break anything, I can try to change that




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org