You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "msugar (via GitHub)" <gi...@apache.org> on 2025/06/16 23:43:50 UTC
[PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
msugar opened a new pull request, #35315:
URL: https://github.com/apache/beam/pull/35315
This blog post shares my experience building Anomaflow, an Apache Beam-based anomaly detection pipeline, during the 2025 Beam College Hackathon. I placed 3rd in the competition and wanted to share my learnings and tips with the community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35315:
URL: https://github.com/apache/beam/pull/35315#issuecomment-2978552521
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35315:
URL: https://github.com/apache/beam/pull/35315#issuecomment-2980583264
Running whitespacelint...
./website/www/site/content/en/blog/beam-college-2025-anomaflow.md:38:86: W201 Trailing whitespace
./website/www/site/content/en/blog/beam-college-2025-anomaflow.md:39:91: W201 Trailing whitespace
Command exited with non-zero status 1
./website/www/site/content/en/blog/beam-college-2025-anomaflow.md:40:66: W201 Trailing whitespace
./website/www/site/content/en/blog/beam-college-2025-anomaflow.md:42:47: W201 Trailing whitespace
./website/www/site/content/en/blog/beam-college-2025-anomaflow.md:43:41: W201 Trailing whitespace
====================================================================================================
129.94user 25.98system 2:38.17elapsed 98%CPU (0avgtext+0avgdata 12288maxresident)k
Whitespacelint has found above issues.
112inputs+0outputs (2major+2301528minor)pagefaults 0swaps
You can open the file with vim and remove trailing whitespaces by:
vim -es -c ":%s/\s\+$//e" -c ":wq" "./website/www/site/content/en/blog/beam-college-2025-anomaflow.md"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35315:
URL: https://github.com/apache/beam/pull/35315#issuecomment-2981462644
Assigning reviewers:
R: @damccorm for label website.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35315:
URL: https://github.com/apache/beam/pull/35315#issuecomment-2978616482
Please fix these errors:
ERROR Author 'msugar' is not defined. Please add them to 'website/www/site/data/authors.yml'
For your MD file, please add the Apache license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35315:
URL: https://github.com/apache/beam/pull/35315#issuecomment-2978628022
Your current blog post follows a good narrative flow. I also suggest some structural changes, for example,
1. **Introduction: The Spark of an Idea**
* Start by setting the scene: you participated in the Beam College Hackathon ([source](https://github.com/apache/beam/pull/35315/files?content_ref=in+the+2025+edition+of+the+beam+college+hackathon+i+had+the+opportunity+to+work+on+a+personal+project+i+called+anomaflow+an+anomaly)).
* Briefly introduce your project, Anomaflow, and its core purpose: an anomaly detection pipeline using Apache Beam and Google Cloud Dataflow ([source](https://github.com/apache/beam/pull/35315/files?content_ref=an+anomaly+detection+pipeline+using+apache+beam+and+google+cloud+dataflow)).
2. **The Project: Building Anomaflow**
* **Goal:** State the primary objective clearly. You already do this well: `The goal was to build a data pipeline capable of processing host telemetry data...` ([source](https://github.com/apache/beam/pull/35315/files?content_ref=the+goal+was+to+build+a+data+pipeline+capable+of+processing+host+telemetry+data+collected+by+bindplane+collectors+which+are+built+on+opentelemetry+i+used+terrafo)).
* **Tech Stack:** Dedicate a small paragraph or a bulleted list to the technologies used (Terraform, Python, Docker, etc.) ([source](https://github.com/apache/beam/pull/35315/files?content_ref=i+used+terraform+to+provision+all+the+gcp+infrastructure+and+python+for+the+code+packaging+it+with+docker+for+deployment)). This is great for readers interested in the technical implementation.
3. **The Hackathon Journey: From Streaming Vision to Batch Reality**
* This section can tell the story of the project's evolution during the hackathon.
* **The Initial Vision:** Describe your original, ambitious goal of a fully streaming pipeline triggered by Pub/Sub notifications ([source](https://github.com/apache/beam/pull/35315/files?content_ref=initially+i+wanted+to+build+a+fully+streaming+solution+where+the+pipeline+would+be+triggered+by+gcs+notifications+published+to+pub+sub+with+output+written+to+a+bigquery+table)).
* **The Pivot:** Explain the practical decision to scope down to a batch pipeline. Framing this as a pragmatic choice made while working solo adds a relatable, human element to the technical narrative ([source](https://github.com/apache/beam/pull/35315/files?content_ref=but+since+i+was+working+solo+i+had+to+scope+it+down+to+a+batch+pipeline+that+reads+from+and+writes+to+a+gcs+bucket)).
* **Key Learnings:** Mention what you gained from the experience, such as the freedom to experiment and iterate quickly ([source](https://github.com/apache/beam/pull/35315/files?content_ref=although+i+already+had+some+real+life+experience+using+apache+beam+the+hackathon+gave+me+the+perfect+excuse+to+experiment+freely+and+iterate+quickly)).
4. **What's Next for Anomaflow?**
* Create a forward-looking section where you discuss your plans to continue developing the project towards your original vision and explore new ideas in cybersecurity analytics ([source](https://github.com/apache/beam/pull/35315/files?content_ref=i+plan+to+continue+developing+anomaflow+to+reach+my+original+vision+and+even+use+it+as+the+foundation+for+new+ideas+i+ve+been+exploring+around+telemetry+processing+and+streaming+analytics+in+cybersecurity)).
5. **Tips for Future Participants**
* Your existing section with tips is excellent and can stand on its own. It provides direct value to prospective attendees ([source](https://github.com/apache/beam/pull/35315/files?content_ref=tips+for+future+participants)).
6. **Conclusion**
* Briefly wrap up your experience and your role at TELUS ([source](https://github.com/apache/beam/pull/35315/files?content_ref=i+m+currently+a+software+architect+and+data+engineer+at+telus+where+i+work+in+the+cybersecurity+analytics+and+software+engineering+team)).
* End with your encouraging closing line and signature ([source](https://github.com/apache/beam/pull/35315/files?content_ref=looking+forward+to+seeing+what+others+build+next+year)).
If possible, you could link your demo slides or videos. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Blog: My Experience at Beam College 2025 (3rd Place Hackathon Winner) [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #35315:
URL: https://github.com/apache/beam/pull/35315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org