You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Ankur Goenka <go...@google.com> on 2018/09/01 00:46:56 UTC

Enhancing Environment Proto to support Docker and Process Environments.

Hi,

We recently added the ProceessEnvironment which uses forked process instead
of Docker container to run SDKHarness.
But the current Environment proto does not have a well defined structure to
represent a process.
Current Proto:

message Environment {

  // (Required) The URL of a container
  //
  // TODO: reconcile with Fn API's DockerContainer structure by
  // adding adequate metadata to know how to interpret the container
  string url = 1;
}

I am planning to enhance the proto to support different types of
environments (Docker and Process for now).
Here is the new proposed proto.

message Environment {

  message Docker {
    // (Required) URL for the container image.
    string url = 1;
    // Arguments to the container.
    repeated string arguments = 2;
    // Environment variable for the container.
    repeated string environments = 3;
  }

  message Process {
    // (Required) Name of the executable.
    string executable = 1;
    // Arguments to the process.
    repeated string arguments = 2;
    // Environment variable for the process.
    repeated string environments = 3;
  }

  // (Required) The unique id of the environment.
  string id = 1;

  // (Required) Environment configuration.
  oneof config {
    Docker docker_config = 2;
    Process process_config = 3;
  }
}

Please let me know your thoughts.

Thanks,
Ankur

Re: Enhancing Environment Proto to support Docker and Process Environments.

Posted by Ankur Goenka <go...@google.com>.

Thanks for identifying that.
It makes sense to continue it over there as there are already enough
suggestions.

On Fri, Aug 31, 2018 at 6:00 PM Henning Rohde <he...@google.com> wrote:

> Hi Ankur,
>
> There is a separate related thread: "Process JobBundleFactory for portable
> runner". It also contains a concrete suggestion for Environment proto
> changes to accommodate docker and process based on a urn/payload structure
> and a but of discussion.
>
> We should either continue the discussion there or bring the two proposals
> together here. What do you think?
>
> Thanks,
>  Henning
>
> On Fri, Aug 31, 2018 at 5:47 PM Ankur Goenka <go...@google.com> wrote:
>
>> Hi,
>>
>> We recently added the ProceessEnvironment which uses forked process
>> instead of Docker container to run SDKHarness.
>> But the current Environment proto does not have a well defined structure
>> to represent a process.
>> Current Proto:
>>
>> message Environment {
>>
>>   // (Required) The URL of a container
>>   //
>>   // TODO: reconcile with Fn API's DockerContainer structure by
>>   // adding adequate metadata to know how to interpret the container
>>   string url = 1;
>> }
>>
>> I am planning to enhance the proto to support different types of
>> environments (Docker and Process for now).
>> Here is the new proposed proto.
>>
>> message Environment {
>>
>>   message Docker {
>>     // (Required) URL for the container image.
>>     string url = 1;
>>     // Arguments to the container.
>>     repeated string arguments = 2;
>>     // Environment variable for the container.
>>     repeated string environments = 3;
>>   }
>>
>>   message Process {
>>     // (Required) Name of the executable.
>>     string executable = 1;
>>     // Arguments to the process.
>>     repeated string arguments = 2;
>>     // Environment variable for the process.
>>     repeated string environments = 3;
>>   }
>>
>>   // (Required) The unique id of the environment.
>>   string id = 1;
>>
>>   // (Required) Environment configuration.
>>   oneof config {
>>     Docker docker_config = 2;
>>     Process process_config = 3;
>>   }
>> }
>>
>> Please let me know your thoughts.
>>
>> Thanks,
>> Ankur
>>
>>
>>

Re: Enhancing Environment Proto to support Docker and Process Environments.

Posted by Henning Rohde <he...@google.com>.

Hi Ankur,

There is a separate related thread: "Process JobBundleFactory for portable
runner". It also contains a concrete suggestion for Environment proto
changes to accommodate docker and process based on a urn/payload structure
and a but of discussion.

We should either continue the discussion there or bring the two proposals
together here. What do you think?

Thanks,
 Henning

On Fri, Aug 31, 2018 at 5:47 PM Ankur Goenka <go...@google.com> wrote:

> Hi,
>
> We recently added the ProceessEnvironment which uses forked process
> instead of Docker container to run SDKHarness.
> But the current Environment proto does not have a well defined structure
> to represent a process.
> Current Proto:
>
> message Environment {
>
>   // (Required) The URL of a container
>   //
>   // TODO: reconcile with Fn API's DockerContainer structure by
>   // adding adequate metadata to know how to interpret the container
>   string url = 1;
> }
>
> I am planning to enhance the proto to support different types of
> environments (Docker and Process for now).
> Here is the new proposed proto.
>
> message Environment {
>
>   message Docker {
>     // (Required) URL for the container image.
>     string url = 1;
>     // Arguments to the container.
>     repeated string arguments = 2;
>     // Environment variable for the container.
>     repeated string environments = 3;
>   }
>
>   message Process {
>     // (Required) Name of the executable.
>     string executable = 1;
>     // Arguments to the process.
>     repeated string arguments = 2;
>     // Environment variable for the process.
>     repeated string environments = 3;
>   }
>
>   // (Required) The unique id of the environment.
>   string id = 1;
>
>   // (Required) Environment configuration.
>   oneof config {
>     Docker docker_config = 2;
>     Process process_config = 3;
>   }
> }
>
> Please let me know your thoughts.
>
> Thanks,
> Ankur
>
>
>