You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by StephanEwen <gi...@git.apache.org> on 2015/07/15 11:28:53 UTC

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

GitHub user StephanEwen opened a pull request:

    https://github.com/apache/flink/pull/913

    [FLINK-2363] [docs] First part of internals - complete job execution overview

    I found that many people do not understand the model behind Flink.
    
    I started drafting/outlining a documentation page that describes end-to-end what happens between the call to `execute()` and the program result being returned or written.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StephanEwen/incubator-flink docs

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #913
    
----
commit c8c562078d917d18b4c0d6671fe9905390550cb5
Author: Stephan Ewen <se...@apache.org>
Date:   2015-07-14T19:15:41Z

    [FLINK-2363] [docs] First part of internals - complete job execution overview.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-123590116
  
    @fhueske Naa, I just want to see it finished. So I'll probably have to write some other parts myself but it would be cool if you could contribute this part.
    
    I'll open my own branch and then people can do pull requests against that, I will try to write the stuff myself or delegate to others for stuff that I don't know very well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-123472145
  
    Hi @aljoscha, I haven't started and could do it but you can also give it a shot if you'd like to :smile: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121611710
  
    I'll fix the typos and update this branch...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121563616
  
    I see, thanks for the effort!
    I'll take care of the *Batch API: DataSets to JobGraph* section.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/913#discussion_r34662306

--- Diff: docs/internals/through_stack.md ---
@@ -0,0 +1,181 @@
+---
+title: "From Program to Result: A Dive through Stack and Execution"
+---
+
+
+This page explains what happens when you execute a Flink program (streaming or streaming).
+It covers the complete life cycle, from <emph>client-side pre-flight</emph>, to <emph>JobManager</emph>,
+and <emph>TaskManager</emph>.
+
+Please refer to the [Overview Page](../index.html) for a high-level overview of the processes and the stack.
+
+* This will be replaced by the TOC
+{:toc}
+
+
+## Batch API: DataSets to JobGraph
+
+## Streaming API: DataStreams to JobGraph
+
+## Client: Submitting a Job, Receiving Statistics & Results
+
+
+
+## JobManager: Receiving and Starting a Job
+
+parallel operator instance
+
+execution attempts
+
+
+## TaskManager: Running Tasks
+
+The *{% gh_link flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala "TaskManager" %}*
+is the worker process in Flink. It runs the *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java "Tasks" %}*,
+which execute a parallel operator instance.
+
+The TaskManager itself is an Akka actor and has many utility components, such as:
+
+ - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java "NetworkEnvironment" %}*, which takes care of all data exchanges (streamed and batched) between TaskManagers. Cached data sets in Flink are also cached streams, to the network environment is responsible for that as well.
+
+ - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/memorymanager/MemoryManager.java "MemoryManager" %}*, which governs the memory for sorting, hashing, and in-operator data caching.
+
+ - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/IOManager.java "I/O Manager" %}*, which governs the memory for sorting, hashing, and in-operator data caching.
+
+ - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/LibraryCacheManager.java "Library Cache" %}*, which gives access to JAR files needed by tasks.
+
+When the TaskManager runs tasks, it does not know anything about the task's role in the dataflow. The TaskManager only knows the tasks itself and streams that the task interacts with.
+Any form of cross-task coordination must go through the JobManager.
+
+The execution of a Task begins when the TaskManager receives the *SubmitTask* message. The message contains the
+*{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/deployment/TaskDeploymentDescriptor.java "TaskDeploymentDescriptor" %}*. This descriptor defines everything
+a task needs to know:
+
+ - The unique task {% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionAttemptID.java "Execution ID" %}.
+ - The name of the executable code class of the task (batch operator, stream operator, iterative operator, ...)
+ - The IDs of the streams that the task reads. If not all streams are ready yet, some are set to "pending".
+ - The IDs of the streams that the task produces.
+ - The configuration of the executable code. This contains the actual operator type (mapper, join, stream window, ...) and the user code, as a serialized closure (MapFunction, JoinFunction, ...)
+ - The hashes if the libraries (JAR files) that the code needs.
+ - ...
+
+The TaskManager creates the *Task object* from the deployment descriptor, registers the task internally under its ID, and spawns a thread that will execute the task.
+After that, the TaskManager is done, and the Task runs by itself, reporting its status to the TaskManager as it progresses or fails.
+
+The Task executes in the following stages:
+
+ 1. It takes the hashes of the required JAR files and makes sure these files are in the library cache. If they are not yet there, the library cache downloads them from the JobManager's BLOB server. This operation may take a while, if the JAR files are very large (many libraries). Note that this only needs to be done for the first Task of a program on each TaskManager. All successive Tasks should find the JAR files cached.
+
+ 2. It creates a classloader from these JAR files. The classloader is called the *user-code classloader* and is used to resolve all classes that can be user-defined.
+
+ 3. It registers its consumed and produces data streams (input and output) at the network environment. This reserves resources for the Task by creating the buffer pools that are used for data exchange.
+
+ 4. In case the task uses state of a checkpoint (a streaming task that is restarted after a failure), it restores this state. This may involve fetching the state from remote storage, depending on where the state was stored.
+
+ 5. The Task switches to *"RUNNING"* and notifies the TaskManager of that progress. The TaskManager in turn sends a {% gh_link /flink-runtime/src/main/scala/org/apache/flink/runtime/messages/TaskMessages.scala#L142 "UpdateTaskExecutionState" %} actor message to the JobManager, to notify it of the progress.
+
+ 6. The Task invokes the executable code that was configured. This code is usually generic and only differentiates between coarse classes of operations (e.g., {% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/operators/RegularPactTask.java "Plain Batch Task" %} , {% gh_link flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java "Streaming Operator" %}, or {% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/iterative/task/IterationIntermediatePactTask.java "Iterative Batch Task" %}). Internally, these generic operators instantiate the specific operator code (e.g., map task, join task, window reducer, ...) and the user functions and executes them.
+
+ 7. If the Task was deployed before all iof its inputs were available (early deployment), the Task receives updates on those newly available streams.
--- End diff --

"iof" -> "of"

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121559138
  
    The text for the task manager execution looks good except for some minor things. I did not check the SVG. 
    
    If you want to put it out like that (incomplete sections), I would add a "Work in progress" in the empty sections to make clear that these gaps will be filled.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-123304832
  
    @fhueske Are you still available for writing the section about the batch API? Or is this void now due to recent events? :smile_cat: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121562249
  
    Thanks for the checks. So many typos ;-)
    
    I opened this pull request as WIP and would encourage other committers to jump in. I have been doing miles of internal docs, website, and design documents. Would be great if someone else finished this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/913#discussion_r34661467

to stream or not to stream? ;-)

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/913#discussion_r34661865
  
    --- Diff: docs/internals/through_stack.md ---
    @@ -0,0 +1,181 @@
    +---
    +title:  "From Program to Result: A Dive through Stack and Execution"
    +---
    +<!--
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +
    +  http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing,
    +software distributed under the License is distributed on an
    +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +KIND, either express or implied.  See the License for the
    +specific language governing permissions and limitations
    +under the License.
    +-->
    +
    +This page explains what happens when you execute a Flink program (streaming or streaming).
    +It covers the complete life cycle, from <emph>client-side pre-flight</emph>, to <emph>JobManager</emph>,
    +and <emph>TaskManager</emph>.
    +
    +Please refer to the [Overview Page](../index.html) for a high-level overview of the processes and the stack.
    +
    +* This will be replaced by the TOC
    +{:toc}
    +
    +
    +## Batch API: DataSets to JobGraph
    +
    +## Streaming API: DataStreams to JobGraph
    +
    +## Client: Submitting a Job, Receiving Statistics & Results
    +
    +
    +
    +## JobManager: Receiving and Starting a Job
    +
    +parallel operator instance
    +
    +execution attempts
    +
    +
    +## TaskManager: Running Tasks
    +
    +The *{% gh_link flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala "TaskManager" %}*
    +is the worker process in Flink. It runs the *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java "Tasks" %}*,
    +which execute a parallel operator instance.
    +
    +The TaskManager itself is an Akka actor and has many utility components, such as:
    +
    + - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java "NetworkEnvironment" %}*, which takes care of all data exchanges (streamed and batched) between TaskManagers. Cached data sets in Flink are also cached streams, to the network environment is responsible for that as well.
    +
    + - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/memorymanager/MemoryManager.java "MemoryManager" %}*, which governs the memory for sorting, hashing, and in-operator data caching.
    +
    + - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/IOManager.java "I/O Manager" %}*, which governs the memory for sorting, hashing, and in-operator data caching.
    --- End diff --
    
    Copy-Paste from MemoryManager


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by rmetzger <gi...@git.apache.org>.

Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121617207
  
    Big +1 to document this properly!
    
    
    The PR contains headlines without content. I would either remove these headlines before merging or merge the PR once its done. 
    (By the coding guidelines, WIP PR's are not allowed anyways)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #913: [FLINK-2363] [docs] First part of internals - compl...

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen closed the pull request at:

    https://github.com/apache/flink/pull/913


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/913#issuecomment-121618152
  
    @rmetzger  You are right. This may be different as it is not my personal WIP, but a WIP that I'd like to hand over...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #913: [FLINK-2363] [docs] First part of internals - complete job...

Posted by coveralls <gi...@git.apache.org>.

Github user coveralls commented on the issue:

    https://github.com/apache/flink/pull/913
  
    
    [![Coverage Status](https://coveralls.io/builds/10937729/badge)](https://coveralls.io/builds/10937729)
    
    Changes Unknown when pulling **e80aa842fbaab611cd889ae35b6c117e54a2e80d on StephanEwen:docs** into ** on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/913#discussion_r34661812
  
    --- Diff: docs/internals/through_stack.md ---
    @@ -0,0 +1,181 @@
    +---
    +title:  "From Program to Result: A Dive through Stack and Execution"
    +---
    +<!--
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +
    +  http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing,
    +software distributed under the License is distributed on an
    +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +KIND, either express or implied.  See the License for the
    +specific language governing permissions and limitations
    +under the License.
    +-->
    +
    +This page explains what happens when you execute a Flink program (streaming or streaming).
    +It covers the complete life cycle, from <emph>client-side pre-flight</emph>, to <emph>JobManager</emph>,
    +and <emph>TaskManager</emph>.
    +
    +Please refer to the [Overview Page](../index.html) for a high-level overview of the processes and the stack.
    +
    +* This will be replaced by the TOC
    +{:toc}
    +
    +
    +## Batch API: DataSets to JobGraph
    +
    +## Streaming API: DataStreams to JobGraph
    +
    +## Client: Submitting a Job, Receiving Statistics & Results
    +
    +
    +
    +## JobManager: Receiving and Starting a Job
    +
    +parallel operator instance
    +
    +execution attempts
    +
    +
    +## TaskManager: Running Tasks
    +
    +The *{% gh_link flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala "TaskManager" %}*
    +is the worker process in Flink. It runs the *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java "Tasks" %}*,
    +which execute a parallel operator instance.
    +
    +The TaskManager itself is an Akka actor and has many utility components, such as:
    +
    + - The *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java "NetworkEnvironment" %}*, which takes care of all data exchanges (streamed and batched) between TaskManagers. Cached data sets in Flink are also cached streams, to the network environment is responsible for that as well.
    --- End diff --
    
    Last sentence should be rephrased.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2363] [docs] First part of internals - ...

Posted by fhueske <gi...@git.apache.org>.

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/913#discussion_r34661712
  
    --- Diff: docs/internals/through_stack.md ---
    @@ -0,0 +1,181 @@
    +---
    +title:  "From Program to Result: A Dive through Stack and Execution"
    +---
    +<!--
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +
    +  http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing,
    +software distributed under the License is distributed on an
    +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +KIND, either express or implied.  See the License for the
    +specific language governing permissions and limitations
    +under the License.
    +-->
    +
    +This page explains what happens when you execute a Flink program (streaming or streaming).
    +It covers the complete life cycle, from <emph>client-side pre-flight</emph>, to <emph>JobManager</emph>,
    +and <emph>TaskManager</emph>.
    +
    +Please refer to the [Overview Page](../index.html) for a high-level overview of the processes and the stack.
    +
    +* This will be replaced by the TOC
    +{:toc}
    +
    +
    +## Batch API: DataSets to JobGraph
    +
    +## Streaming API: DataStreams to JobGraph
    +
    +## Client: Submitting a Job, Receiving Statistics & Results
    +
    +
    +
    +## JobManager: Receiving and Starting a Job
    +
    +parallel operator instance
    +
    +execution attempts
    +
    +
    +## TaskManager: Running Tasks
    +
    +The *{% gh_link flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala "TaskManager" %}*
    +is the worker process in Flink. It runs the *{% gh_link flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java "Tasks" %}*,
    +which execute a parallel operator instance.
    --- End diff --
    
    "a parallel operator instance." -> "parallel operators instances."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---