[jira] [Updated] (SAMZA-438) Pass config via HTTP

["Chris Riccomini (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Riccomini updated SAMZA-438:
----------------------------------
    Attachment: SAMZA-438-0.patch

Changes made:

# Removed environment variable compression (SAMZA-337), since we're passing via HTTP now.
# Converted YARN's WebAppServer to a generic HttpServer in samza-core.
# Wrote JobServlet, which serves a job's config, task:SSP mapping, and task:changelog partition mapping via HTTP JSON.
# Updated ShellCommandBuilder/CommandBuilder to set an HTTP URL environment variable, rather than config, task:SSP mapping, and task:changelog partition mapping.
# Updated ProcessJob, ThreadJob, and the AM code to run the HttpServer/JobServlet, and set the HTTP URL environment variable when starting SamzaContainers.
# Update SamzaContainer to fetch config, task:SSP mapping, and task:changelog partition mapping using HTTP URL environment variable.
# Changed container name to container ID in SamzaContainerContext, and run-\*.sh. Kept legacy "samza.container.name" system property, so we're backwards compatible with log4j.properties files that refer to it (hello-samza).
# Wrote a JsonHelpers class to help with all the terrible back bending we have to do to make Scala work with Jackson.
# Updated Util to remove the compress methods, and add a helper "read" method for reading from HTTP URLs.

Remaining items in this ticket:

# Write tests.

Tickets I'd like to open as follow ons:

# Remove TaskNamesToSystemStreamPartitions, and convert to a proper job/container/task data model. Make the data model Java based, and use Jackson annotations.
# Convert the HttpServer to be a proper Jetty/Jersey/Jackson server that uses the properly defined data model to serve its content.


> Pass config via HTTP
> --------------------
>
>                 Key: SAMZA-438
>                 URL: https://issues.apache.org/jira/browse/SAMZA-438
>             Project: Samza
>          Issue Type: Sub-task
>            Reporter: Chris Riccomini
>            Assignee: Chris Riccomini
>         Attachments: SAMZA-438-0.patch
>
>
> SAMZA-348 has a detailed design proposal on how we can configure Samza via a stream. Part of this work involves converting the SamzaContainer to retrieve its information via an HTTP/JSON request, rather than from JSON encoded environment variables.
> The three items that we'll need to serve via a "job coordinator" will be the job's config, the job's container:task:SSP mappings, and the job's task:changelog partition mappings.
> We'll also need to assign each container a unique ID, so that it can retrieve its task:SSP mappings (and thus, its task:changelog partition mappings).
> This ticket does not encompass creating a "job coordinator". Instead, we'll just start the HTTP server in three places: the ThreadJob, the ProcessJob, and the YARN AM. In all three cases, a URL will be set via an environment variable, which the SamzaContainer will receive, and use to retrieve its information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)