You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/09/15 21:21:58 UTC
[GitHub] [beam] KevinGG opened a new pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
KevinGG opened a new pull request #15518:
URL: https://github.com/apache/beam/pull/15518
An example notebook demonstrating the usage of the beam_sql magic.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.77% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50419 +39
- Misses 9750 9768 +18
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...b982fd5](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
> :exclamation: Current head b982fd5 differs from pull request most recent head e96e00a. Consider uploading reports for the commit e96e00a to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.77% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50419 +39
- Misses 9750 9768 +18
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...e96e00a](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a87a3c8) into [master](https://codecov.io/gh/apache/beam/commit/6947fa2136e2d74e51de8347f852aa85387d4735?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6947fa2) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.58%`.
> :exclamation: Current head a87a3c8 differs from pull request most recent head 3b09cd4. Consider uploading reports for the commit 3b09cd4 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.77% 83.75% -0.02%
==========================================
Files 444 444
Lines 60255 60240 -15
==========================================
- Hits 50479 50457 -22
- Misses 9776 9783 +7
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <53.16%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <62.50%> (-5.59%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (ø)` | |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (ø)` | |
| ... and [45 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...3b09cd4](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b09cd4) into [master](https://codecov.io/gh/apache/beam/commit/6947fa2136e2d74e51de8347f852aa85387d4735?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6947fa2) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.58%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.77% 83.75% -0.02%
==========================================
Files 444 444
Lines 60255 60318 +63
==========================================
+ Hits 50479 50520 +41
- Misses 9776 9798 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <20.00%> (+0.15%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <53.16%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <62.50%> (-5.59%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (ø)` | |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (ø)` | |
| ... and [45 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...3b09cd4](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.76% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50418 +38
- Misses 9750 9769 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...b982fd5](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG merged pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG merged pull request #15518:
URL: https://github.com/apache/beam/pull/15518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713260530
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
Review comment:
As a first time reader, the word "magic" confuses me: why and how this is a magic? Magic of what? Is it a workaround of something or is it a SQL on some magic data?
May be helpful to add a brief description here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713294459
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
+ "\n",
+ "Note that we have **not** explicitly created a Beam pipeline. We get a PCollection because the `beam_sql` magic always **implicitly creates** a pipeline to execute your SQL query. To hold the elements with each field's type info, Beam automatically creates a schema as the element_type for the created PCollection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "05d2b688",
+ "metadata": {},
+ "source": [
+ "To introspect the data again with more knobs, we can use `show`."
Review comment:
Change to use `you`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713254441
##########
File path: sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
##########
@@ -238,7 +239,13 @@ def continuous_update_display(): # pylint: disable=unused-variable
def visualize_computed_pcoll(
- pcoll_name: str, pcoll: beam.pvalue.PCollection) -> None:
+ pcoll_name: str,
+ pcoll: beam.pvalue.PCollection,
+ max_n: int,
+ max_duration_secs: float,
+ dynamic_plotting_interval: Optional[int] = None,
+ include_window_info: bool = False,
+ display_facets: bool = False) -> None:
Review comment:
Yes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.76% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50418 +38
- Misses 9750 9769 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...b982fd5](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.76% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50418 +38
- Misses 9750 9769 +19
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...b982fd5](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713291396
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
+ "\n",
+ "Note that we have **not** explicitly created a Beam pipeline. We get a PCollection because the `beam_sql` magic always **implicitly creates** a pipeline to execute your SQL query. To hold the elements with each field's type info, Beam automatically creates a schema as the element_type for the created PCollection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "05d2b688",
+ "metadata": {},
+ "source": [
+ "To introspect the data again with more knobs, we can use `show`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "89220b13",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from apache_beam.runners.interactive import interactive_beam as ib\n",
+ "ib.show(query1_data)\n",
+ "# Uncomment below to set more args.\n",
+ "# ib.show(query1_data, visualize_data=True, include_window_info=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6faa7df8",
+ "metadata": {},
+ "source": [
+ "To materialize the PCollection into a pandas DataFrame object, we can use `collect`."
Review comment:
Got it, adding https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html#dataframe as the link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/a7b706cb9d1e84709f89fe98d1dda94d4eb1243b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7b706c) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head b982fd5 differs from pull request most recent head a87a3c8. Consider uploading reports for the commit a87a3c8 to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
=======================================
Coverage 83.76% 83.77%
=======================================
Files 444 444
Lines 60189 60187 -2
=======================================
+ Hits 50416 50419 +3
+ Misses 9773 9768 -5
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `92.30% <0.00%> (-3.30%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `92.09% <0.00%> (-0.05%)` | :arrow_down: |
| [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.86% <0.00%> (+0.13%)` | :arrow_up: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.51% <0.00%> (+0.24%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `89.86% <0.00%> (+1.38%)` | :arrow_up: |
| [.../python/apache\_beam/transforms/periodicsequence.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wZXJpb2RpY3NlcXVlbmNlLnB5) | `98.36% <0.00%> (+1.63%)` | :arrow_up: |
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.02% <0.00%> (+2.23%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `95.12% <0.00%> (+2.43%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...a87a3c8](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713265671
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
Review comment:
I'll move the explanation to the very beginning, immediately after mentioning the `beam_sql` magic.
> `beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html) ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713273996
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
+ "\n",
+ "Note that we have **not** explicitly created a Beam pipeline. We get a PCollection because the `beam_sql` magic always **implicitly creates** a pipeline to execute your SQL query. To hold the elements with each field's type info, Beam automatically creates a schema as the element_type for the created PCollection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "05d2b688",
+ "metadata": {},
+ "source": [
+ "To introspect the data again with more knobs, we can use `show`."
Review comment:
This doc mixes the usage of you and we, sometimes is "you can do something" and sometimes is "we can do something", consider making it consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b09cd4) into [master](https://codecov.io/gh/apache/beam/commit/6947fa2136e2d74e51de8347f852aa85387d4735?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6947fa2) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.58%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.77% 83.75% -0.02%
==========================================
Files 444 444
Lines 60255 60318 +63
==========================================
+ Hits 50479 50520 +41
- Misses 9776 9798 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <20.00%> (+0.15%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <53.16%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <62.50%> (-5.59%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (ø)` | |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (ø)` | |
| ... and [45 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...3b09cd4](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-924285935
Run PythonDocker PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713264952
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
Review comment:
I think this sentence is better placed when you mentioned "magic" above (i.e., first let's load the beam_sql magic), which gives reader more context about magic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713296466
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
Review comment:
Enclosing literal words with `_` with backticks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713248395
##########
File path: sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
##########
@@ -238,7 +239,13 @@ def continuous_update_display(): # pylint: disable=unused-variable
def visualize_computed_pcoll(
- pcoll_name: str, pcoll: beam.pvalue.PCollection) -> None:
+ pcoll_name: str,
+ pcoll: beam.pvalue.PCollection,
+ max_n: int,
+ max_duration_secs: float,
+ dynamic_plotting_interval: Optional[int] = None,
+ include_window_info: bool = False,
+ display_facets: bool = False) -> None:
Review comment:
Can you add some comments for these new parameters? I have to look at the implementation to understand their meanings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b982fd5) into [master](https://codecov.io/gh/apache/beam/commit/f5afff17de0898bf0d2116e7757d89315f508cad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f5afff1) will **decrease** coverage by `0.01%`.
> The diff coverage is `89.14%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.78% 83.77% -0.02%
==========================================
Files 443 444 +1
Lines 60130 60187 +57
==========================================
+ Hits 50380 50419 +39
- Misses 9750 9768 +18
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (-0.62%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.00% <20.00%> (+0.05%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (-0.21%)` | :arrow_down: |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `57.46% <63.15%> (+2.76%)` | :arrow_up: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (-2.43%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (+0.84%)` | :arrow_up: |
| [...n/apache\_beam/runners/interactive/caching/reify.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL3JlaWZ5LnB5) | `96.66% <96.66%> (ø)` | |
| [sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=) | `91.69% <100.00%> (+0.02%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <100.00%> (+0.21%)` | :arrow_up: |
| [...beam/runners/interactive/background\_caching\_job.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9iYWNrZ3JvdW5kX2NhY2hpbmdfam9iLnB5) | `96.55% <100.00%> (-0.03%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1723745...b982fd5](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b09cd4) into [master](https://codecov.io/gh/apache/beam/commit/6947fa2136e2d74e51de8347f852aa85387d4735?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6947fa2) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.58%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.77% 83.75% -0.02%
==========================================
Files 444 444
Lines 60255 60318 +63
==========================================
+ Hits 50479 50520 +41
- Misses 9776 9798 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <20.00%> (+0.15%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <53.16%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <62.50%> (-5.59%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (ø)` | |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (ø)` | |
| ... and [45 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...3b09cd4](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713275461
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
+ "\n",
+ "Note that we have **not** explicitly created a Beam pipeline. We get a PCollection because the `beam_sql` magic always **implicitly creates** a pipeline to execute your SQL query. To hold the elements with each field's type info, Beam automatically creates a schema as the element_type for the created PCollection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "05d2b688",
+ "metadata": {},
+ "source": [
+ "To introspect the data again with more knobs, we can use `show`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "89220b13",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from apache_beam.runners.interactive import interactive_beam as ib\n",
+ "ib.show(query1_data)\n",
+ "# Uncomment below to set more args.\n",
+ "# ib.show(query1_data, visualize_data=True, include_window_info=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6faa7df8",
+ "metadata": {},
+ "source": [
+ "To materialize the PCollection into a pandas DataFrame object, we can use `collect`."
Review comment:
Maybe add a link to pandas DataFrame object? I have no idea what is that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andyxiexu commented on a change in pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
andyxiexu commented on a change in pull request #15518:
URL: https://github.com/apache/beam/pull/15518#discussion_r713271781
##########
File path: sdks/python/apache_beam/runners/interactive/examples/Run Beam SQL with beam_sql magic.ipynb
##########
@@ -0,0 +1,548 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a999c0fd",
+ "metadata": {},
+ "source": [
+ "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "<!--\n",
+ " Licensed to the Apache Software Foundation (ASF) under one\n",
+ " or more contributor license agreements. See the NOTICE file\n",
+ " distributed with this work for additional information\n",
+ " regarding copyright ownership. The ASF licenses this file\n",
+ " to you under the Apache License, Version 2.0 (the\n",
+ " \"License\"); you may not use this file except in compliance\n",
+ " with the License. You may obtain a copy of the License at\n",
+ "\n",
+ " http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ " Unless required by applicable law or agreed to in writing,\n",
+ " software distributed under the License is distributed on an\n",
+ " \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ " KIND, either express or implied. See the License for the\n",
+ " specific language governing permissions and limitations\n",
+ " under the License.\n",
+ "-->\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5562c11a",
+ "metadata": {},
+ "source": [
+ "# Run Beam SQL in notebooks\n",
+ "\n",
+ "[Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/) allows a Beam user to query PCollections with SQL statements. Currently, `InteractiveRunner` does not support `SqlTransform` due to [BEAM-10708](https://issues.apache.org/jira/browse/BEAM-10708). However, a user could use the `beam_sql` magic to run Beam SQL in the notebook and introspect the result. It's a convenient way to validate your queries locally against known/test data sources when prototyping a Beam pipeline with SQL, before productionizing it on remote cluster/services. \n",
+ "\n",
+ "First, let's load the `beam_sql` magic:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8928343a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext apache_beam.runners.interactive.sql.beam_sql_magics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dec39a04",
+ "metadata": {},
+ "source": [
+ "Since SQL support in Beam Python SDK is implemented through xLang external transform, make sure you have below prerequisites:\n",
+ "- Have `docker` installed;\n",
+ "- Have jdk8 or jdk11 installed and $JAVA_HOME set;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa9cede3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!docker image list\n",
+ "!java --version\n",
+ "!echo $JAVA_HOME"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "640a427e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optionally sets the logging level to reduce distraction.\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ab6438a",
+ "metadata": {},
+ "source": [
+ "**Important**: if you're using Beam built from your local source code, additionally:\n",
+ "\n",
+ "- Have the Java expansion service shadowjar built. Go to the root directory of your local beam repo and then execute:\n",
+ " `./gradlew :sdks:java:extensions:sql:expansion-service:shadowJar`;\n",
+ "- Based on your jdk version, pull the docker image `docker pull apache/beam_java11_sdk` or `docker pull apache/beam_java8_sdk`.\n",
+ "- Then tag the image with your current Beam dev version. You can check the dev version under `apache_beam.version.__version__`. For example, if you're using jdk11 and dev version is `x.x.x.dev`, execute `docker image tag apache/beam_java11_sdk:latest apache/beam_java11_sdk:x.x.x.dev`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcf28852",
+ "metadata": {},
+ "source": [
+ "## Query#1 - A simple static query\n",
+ "`beam_sql` is an IPython [custom magic](https://ipython.readthedocs.io/en/stable/config/custommagics.html). If you're not familiar with magics, here are some [built-in examples](https://ipython.readthedocs.io/en/stable/interactive/magics.html).\n",
+ "\n",
+ "The `beam_sql` magic can be used as either a line magic or a cell magic.\n",
+ "We can check its usage by running:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5777a84e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%beam_sql -h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "20f88c5d",
+ "metadata": {},
+ "source": [
+ "We can run a simple SQL query (in Apache Calcite SQL [syntax](https://beam.apache.org/documentation/dsls/sql/calcite/query-syntax/)) to create a [schema-aware PCollection](https://beam.apache.org/documentation/programming-guide/#schemas) from static values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49c70de3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%beam_sql -o query1_data\n",
+ "SELECT CAST(5 AS INT) AS `id`, CAST('foo' AS VARCHAR) AS `str`, CAST(3.14 AS DOUBLE) AS `flt`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6a1c5e9e",
+ "metadata": {},
+ "source": [
+ "The `beam_sql` magic shows you the result of the SQL query.\n",
+ "\n",
+ "It also creates and outputs a PCollection named `query1_data` with element_type like `BeamSchema_...(id: int32, str: str)`.\n",
Review comment:
element_type does not display properly.
Please check other name with '_' in this doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3b09cd4) into [master](https://codecov.io/gh/apache/beam/commit/6947fa2136e2d74e51de8347f852aa85387d4735?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6947fa2) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.58%`.
> :exclamation: Current head 3b09cd4 differs from pull request most recent head a6bb4ae. Consider uploading reports for the commit a6bb4ae to get more accurate results
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.77% 83.75% -0.02%
==========================================
Files 444 444
Lines 60255 60318 +63
==========================================
+ Hits 50479 50520 +41
- Misses 9776 9798 +22
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...che\_beam/runners/interactive/augmented\_pipeline.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9hdWdtZW50ZWRfcGlwZWxpbmUucHk=) | `69.38% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.16% <20.00%> (+0.15%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.50% <40.00%> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <53.16%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <62.50%> (-5.59%)` | :arrow_down: |
| [...ache\_beam/runners/interactive/recording\_manager.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9yZWNvcmRpbmdfbWFuYWdlci5weQ==) | `96.55% <66.66%> (ø)` | |
| [...ache\_beam/runners/interactive/caching/cacheable.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9jYWNoaW5nL2NhY2hlYWJsZS5weQ==) | `97.61% <94.11%> (ø)` | |
| ... and [45 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...a6bb4ae](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a87a3c8) into [master](https://codecov.io/gh/apache/beam/commit/a7b706cb9d1e84709f89fe98d1dda94d4eb1243b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7b706c) will **decrease** coverage by `0.00%`.
> The diff coverage is `60.27%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.76% 83.75% -0.01%
==========================================
Files 444 444
Lines 60189 60240 +51
==========================================
+ Hits 50416 50457 +41
- Misses 9773 9783 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <43.18%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <50.00%> (-5.59%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `95.23% <85.71%> (-0.37%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `86.36% <100.00%> (+0.57%)` | :arrow_up: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `77.77% <100.00%> (+2.24%)` | :arrow_up: |
| [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.86% <0.00%> (+0.13%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
| ... and [6 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...a87a3c8](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-924318848
Squashed the commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920401386
# [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#15518](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a87a3c8) into [master](https://codecov.io/gh/apache/beam/commit/a7b706cb9d1e84709f89fe98d1dda94d4eb1243b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7b706c) will **decrease** coverage by `0.00%`.
> The diff coverage is `60.27%`.
[](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #15518 +/- ##
==========================================
- Coverage 83.76% 83.75% -0.01%
==========================================
Files 444 444
Lines 60189 60240 +51
==========================================
+ Hits 50416 50457 +41
- Misses 9773 9783 +10
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.85% <ø> (ø)` | |
| [...apache\_beam/typehints/native\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL25hdGl2ZV90eXBlX2NvbXBhdGliaWxpdHkucHk=) | `86.57% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `94.20% <ø> (ø)` | |
| [...he\_beam/runners/interactive/sql/beam\_sql\_magics.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvYmVhbV9zcWxfbWFnaWNzLnB5) | `54.54% <43.18%> (-2.92%)` | :arrow_down: |
| [...ython/apache\_beam/runners/interactive/sql/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9zcWwvdXRpbHMucHk=) | `84.00% <50.00%> (-5.59%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/interactive/utils.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS91dGlscy5weQ==) | `95.23% <85.71%> (-0.37%)` | :arrow_down: |
| [...runners/interactive/display/pcoll\_visualization.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3Bjb2xsX3Zpc3VhbGl6YXRpb24ucHk=) | `86.36% <100.00%> (+0.57%)` | :arrow_up: |
| [...pache\_beam/runners/interactive/interactive\_beam.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9iZWFtLnB5) | `77.77% <100.00%> (+2.24%)` | :arrow_up: |
| [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.86% <0.00%> (+0.13%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `95.98% <0.00%> (+0.16%)` | :arrow_up: |
| ... and [6 more](https://codecov.io/gh/apache/beam/pull/15518/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6c1c19d...a87a3c8](https://codecov.io/gh/apache/beam/pull/15518?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920449796
Run Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on pull request #15518: [BEAM-10708] Added an example notebook for beam_sql magic
Posted by GitBox <gi...@apache.org>.
KevinGG commented on pull request #15518:
URL: https://github.com/apache/beam/pull/15518#issuecomment-920392423
R: @rohdesamuel
PTAL, thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org