You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2019/10/07 20:14:00 UTC
[jira] [Work logged] (BEAM-7926) Visualize PCollection with
Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=324584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324584 ]
ASF GitHub Bot logged work on BEAM-7926:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Oct/19 20:13
Start Date: 07/Oct/19 20:13
Worklog Time Spent: 10m
Work Description: KevinGG commented on pull request #9741: [BEAM-7926] Visualize PCollection
URL: https://github.com/apache/beam/pull/9741
1. Added pcoll_visualization module to visualize materialized
PCollection data.
2. A visualization mainly consists of 3 parts: facets-dive,
facets-overview and data-table.
3. The visualization is always async. It has 2 modes: one-shot and
dynamical plotting.
4. One-shot visualization renders the visualization as HTML statically.
5. Dynamical plotting polls materialized data, initializes a
visualization and then updates the visualization continuously.
6. It's the pipeline state, i.e., whether the PCollection
materialization is being updated that decides if the visualization is
dynamical plotting or one-shot. It's not the boundedness of PCollection
that decides the visualization mode. However, when engineering for
Interactive Beam, it's alright to always use the dynamical plotting
mode as long as the InteractiveRunner(s) manages jobs and tracks the
pipeline(job) results correctly. Failing to terminate a visualization
job might cause resource leak.
7. The materialized PCollection data goes through the following process
flow: cache -> element list -> normalization -> dataframe ->
HTMLs/scripts.
8. A visualization is always self-contained. Meaning it controls when
the visualization starts and ends by itself. Once visualized, the
visualization is anchored within the notebook cell's output area. Code
in other cells can affect the visualization in the cell but will not
change the existing visualization's position in the notebook. Multiple
visualize(pcoll) calls will render multiple independent distinct
visualization instances/jobs. Each visualization runs in its own thread
outside of the notebook main thread.
8. Unittest added focusing on the above visualization logic other than
the visualization UI (those should belong to Facets, pandas and jQuery,
we'll trust them work as intended). Integration tests can be added
later.
9. Each visualization job regularly queries interactive environment for
running pipeline(job) results implicitly to determine the default
visualization end condition. It's based on the limitation that within
current interactive environment, at a time, at most one job is in
non-terminated state for an end user defined pipeline instance (in
whatever state). The end user could also use the handle returned by
visualize() to stop() the visualization job manually.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 324584)
Remaining Estimate: 0h
Time Spent: 10m
> Visualize PCollection with Interactive Beam
> -------------------------------------------
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
> Reporter: Ning Kang
> Assignee: Ning Kang
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection with Interactive Beam.
> Say an Interactive Beam pipeline defined as
> p = create_pipeline()
> pcoll = p | 'Transform' >> transform()
> The use can call a single function and get auto-magical charting of the data as materialized pcoll.
> e.g., visualize(pcoll)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)