You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hamilton.apache.org by sk...@apache.org on 2025/06/21 05:49:18 UTC
(hamilton) 02/05: Updates docs to reference Hamilton as Apache Hamilton
This is an automated email from the ASF dual-hosted git repository.
skrawcz pushed a commit to branch update_references
in repository https://gitbox.apache.org/repos/asf/hamilton.git
commit 9960c3f0e98753e46cb099116762746325d8adc4
Author: Stefan Krawczyk <st...@cs.stanford.edu>
AuthorDate: Sun Jun 8 21:04:01 2025 -0700
Updates docs to reference Hamilton as Apache Hamilton
I spot checked this.
---
.github/ISSUE_TEMPLATE/bug_report.md | 2 +-
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
.../HAMILTON_CONTRIB_PR_TEMPLATE.md | 4 +-
CONTRIBUTING.md | 2 +-
README.md | 68 ++++++-------
basics.md | 42 ++++----
contrib/README.md | 14 +--
contrib/docs/blog/2023-11-13-welcome/index.md | 8 +-
contrib/docs/docs/README.md | 20 ++--
contrib/hamilton/contrib/dagworks/author.md | 2 +-
.../contrib/dagworks/conversational_rag/README.md | 6 +-
.../hamilton/contrib/dagworks/faiss_rag/README.md | 6 +-
.../contrib/dagworks/simple_eval_grader/README.md | 2 +-
.../dagworks/translate_to_hamilton/README.md | 2 +-
.../hamilton/contrib/user/elijahbenizzy/author.md | 2 +-
contrib/hamilton/contrib/user/skrawcz/author.md | 2 +-
.../user/skrawcz/customize_embeddings/README.md | 2 +-
.../contrib/user/skrawcz/fine_tuning/README.md | 2 +-
.../contrib/user/zilto/xgboost_optuna/README.md | 2 +-
data_quality.md | 2 +-
decorators.md | 4 +-
dev_tools/language_server/README.md | 6 +-
dev_tools/vscode_extension/README.md | 2 +-
.../resources/walkthrough/4_visitLSP.md | 4 +-
.../walkthrough/5_visitDataflowWebview.md | 2 +-
.../resources/walkthrough/6_tips.md | 2 +-
developer_setup.md | 2 +-
docs/code-comparisons/airflow.rst | 28 +++---
docs/code-comparisons/dagster.rst | 18 ++--
docs/code-comparisons/index.rst | 2 +-
docs/code-comparisons/kedro.rst | 50 +++++-----
docs/code-comparisons/langchain.rst | 58 +++++------
docs/concepts/best-practices/code-organization.rst | 8 +-
docs/concepts/best-practices/common-indices.rst | 4 +-
docs/concepts/best-practices/function-naming.rst | 10 +-
docs/concepts/best-practices/index.rst | 2 +-
docs/concepts/best-practices/loading-data.rst | 8 +-
.../best-practices/migrating-to-hamilton.rst | 20 ++--
.../best-practices/output-immutability.rst | 6 +-
.../using-within-your-etl-system.rst | 6 +-
docs/concepts/builder.rst | 8 +-
docs/concepts/caching.rst | 16 ++--
docs/concepts/driver.rst | 8 +-
docs/concepts/function-modifiers-advanced.rst | 2 +-
docs/concepts/function-modifiers.rst | 24 ++---
docs/concepts/glossary.rst | 8 +-
docs/concepts/index.rst | 2 +-
docs/concepts/materialization.rst | 20 ++--
docs/concepts/node.rst | 32 +++----
docs/concepts/parallel-task.rst | 10 +-
docs/concepts/visualization.rst | 6 +-
docs/get-started/index.rst | 12 +--
docs/get-started/install.md | 6 +-
docs/get-started/learning-resources.md | 106 ++++++++++-----------
docs/get-started/license.rst | 10 +-
docs/get-started/why-hamilton.md | 46 ++++-----
docs/get-started/your-first-dataflow.rst | 8 +-
docs/hamilton-ui/index.rst | 2 +-
docs/hamilton-ui/ui.rst | 32 +++----
docs/hamilton-vscode/language_server.rst | 8 +-
docs/hamilton-vscode/vscode_extension.rst | 14 +--
docs/how-tos/cli-reference.md | 4 +-
docs/how-tos/extensions-autoloading.rst | 10 +-
docs/how-tos/index.rst | 2 +-
docs/how-tos/llm-workflows.md | 8 +-
docs/how-tos/load-data.rst | 4 +-
docs/how-tos/microservice.rst | 2 +-
docs/how-tos/ml-training.rst | 4 +-
docs/how-tos/pre-commit-hooks.md | 16 ++--
docs/how-tos/run-data-quality-checks.rst | 2 +-
docs/how-tos/scale-up.rst | 4 +-
docs/how-tos/use-for-feature-engineering.rst | 50 +++++-----
docs/how-tos/use-in-jupyter-notebook.md | 36 +++----
docs/how-tos/wrapping-driver.rst | 12 +--
docs/integrations/dbt.rst | 8 +-
docs/integrations/dlt/index.md | 36 +++----
docs/integrations/fastapi.md | 24 ++---
docs/integrations/ibis/index.md | 34 +++----
docs/integrations/index.rst | 2 +-
docs/integrations/streamlit.md | 26 ++---
docs/main.md | 42 ++++----
docs/reference/caching/caching-logic.rst | 6 +-
docs/reference/dataflows/index.rst | 2 +-
docs/reference/dataflows/inspect.rst | 2 +-
docs/reference/dataflows/inspect_module.rst | 2 +-
docs/reference/decorators/index.rst | 2 +-
docs/reference/disabling-telemetry.md | 2 +-
docs/reference/drivers/index.rst | 4 +-
docs/reference/graph-adapters/DaskGraphAdapter.rst | 2 +-
docs/reference/graph-adapters/RayGraphAdapter.rst | 2 +-
.../graph-adapters/ThreadPoolFutureAdapter.rst | 2 +-
docs/reference/graph-adapters/index.rst | 2 +-
docs/reference/io/index.rst | 2 +-
docs/reference/lifecycle-hooks/ProgressBar.rst | 2 +-
docs/reference/lifecycle-hooks/RichProgressBar.rst | 2 +-
.../lifecycle-hooks/SlackNotifierHook.rst | 2 +-
docs/reference/lifecycle-hooks/index.rst | 2 +-
examples/LLM_Workflows/GraphRAG/README.md | 2 +-
.../RAG_document_extract_chunk_embed/README.md | 2 +-
examples/LLM_Workflows/README.md | 2 +-
examples/LLM_Workflows/image_telephone/README.md | 6 +-
.../LLM_Workflows/knowledge_retrieval/README.md | 4 +-
.../LLM_Workflows/langchain_comparison/README.md | 8 +-
examples/LLM_Workflows/modular_llm_stack/README.md | 6 +-
.../observability_openllmetry/README.md | 8 +-
examples/LLM_Workflows/pdf_summarizer/README.md | 8 +-
.../pdf_summarizer/run_on_spark/README.md | 8 +-
.../retrieval_augmented_generation/README.md | 8 +-
.../LLM_Workflows/scraping_and_chunking/README.md | 20 ++--
.../scraping_and_chunking/spark/README.md | 8 +-
examples/README.md | 2 +-
examples/airflow/README.md | 18 ++--
examples/async/README.md | 2 +-
examples/aws/glue/README.md | 8 +-
examples/aws/lambda/README.md | 4 +-
examples/aws/sagemaker/README.md | 4 +-
examples/caching/README.md | 2 +-
examples/caching_nodes/README.md | 4 +-
examples/caching_nodes/diskcache_adapter/README.md | 2 +-
examples/cli/DOCS.md | 2 +-
examples/cli/README.md | 6 +-
examples/contrib/README.md | 2 +-
examples/dagster/README.md | 12 +--
examples/dagster/hamilton_code/README.md | 10 +-
examples/dask/README.md | 4 +-
examples/dask/community_demo/README.md | 10 +-
examples/dask/hello_world/README.md | 4 +-
examples/data_loaders/README.md | 4 +-
examples/data_quality/pandera/README.md | 14 +--
examples/data_quality/simple/README.md | 12 +--
examples/dbt/README.md | 26 ++---
examples/dlt/README.md | 6 +-
examples/due_date_probabilities/README.md | 10 +-
examples/experiment_management/README.md | 10 +-
examples/feast/README.md | 14 +--
examples/feast/default_feature_store/README.md | 2 +-
.../README.md | 30 +++---
.../scenario_1/README.md | 4 +-
.../scenario_2/README.md | 2 +-
.../write_once_run_everywhere_blog_post/README.md | 2 +-
.../hamilton-tutorials/mpg-translation/README.md | 8 +-
examples/hamilton_ui/README.md | 22 ++---
examples/hello_world/README.md | 6 +-
examples/ibis/feature_engineering/README.md | 12 +--
examples/ibis/jaffle_shop/README.md | 8 +-
examples/ibisml/README.md | 2 +-
examples/jupyter_notebook_magic/README.md | 6 +-
examples/kedro/README.md | 12 +--
examples/kedro/hamilton-code/README.md | 14 +--
examples/kedro/kedro-plugin/README.md | 2 +-
examples/lineage/README.md | 66 ++++++-------
examples/materialization/README.md | 2 +-
.../datasaver_dataloader_example/README.md | 6 +-
examples/mlflow/README.md | 12 +--
examples/model_examples/README.md | 4 +-
examples/model_examples/modular_example/README.md | 2 +-
examples/model_examples/scikit-learn/README.md | 4 +-
examples/model_examples/time-series/README.md | 4 +-
examples/module_overrides/README.md | 4 +-
examples/numpy/air-quality-analysis/README.md | 6 +-
examples/openlineage/README.md | 4 +-
examples/opentelemetry/README.md | 4 +-
examples/pandas/split-apply-combine/README.md | 2 +-
examples/parallelism/README.md | 2 +-
.../lazy_threadpool_execution/README.md | 2 +-
examples/parallelism/star_counting/README.md | 4 +-
examples/people_data_labs/README.md | 12 +--
examples/plotly/README.md | 2 +-
examples/polars/README.md | 10 +-
examples/polars/lazyframe/README.md | 10 +-
examples/prefect/README.md | 18 ++--
examples/ray/README.md | 6 +-
examples/ray/hello_world/README.md | 4 +-
.../species_distribution_modeling/README.md | 8 +-
examples/scikit-learn/transformer/README.md | 8 +-
examples/snowflake/hamilton_ui/README.md | 12 +--
examples/spark/README.md | 10 +-
examples/spark/pandas_on_spark/README.md | 4 +-
examples/spark/pyspark/README.md | 30 +++---
examples/spark/pyspark_udfs/README.md | 34 +++----
examples/streamlit/README.md | 16 ++--
examples/styling_visualization/README.md | 8 +-
examples/vaex/README.md | 2 +-
examples/validation/static_validator/README.md | 2 +-
hamilton/plugins/README.md | 2 +-
plugin_tests/README.md | 2 +-
ui/README.md | 12 +--
ui/frontend/README.md | 2 +-
ui/sdk/README.md | 22 ++---
writeups/garbage_collection/post.md | 18 ++--
190 files changed, 987 insertions(+), 987 deletions(-)
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index e15f9e89..78183af4 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -1,6 +1,6 @@
---
name: Bug report
-about: Create a bug report to help us improve Hamilton.
+about: Create a bug report to help us improve Apache Hamilton.
title: Bug Report
labels: triage
assignees: ''
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 63a69372..691b4afc 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,6 +1,6 @@
--- PR TEMPLATE INSTRUCTIONS (1) ---
-Looking to submit a Hamilton Dataflow to the sf-hamilton-contrib module? If so go the the `Preview` tab and select the appropriate sub-template:
+Looking to submit a Apache Hamilton Dataflow to the sf-hamilton-contrib module? If so go the the `Preview` tab and select the appropriate sub-template:
* [sf-hamilton-contrib template](?expand=1&template=HAMILTON_CONTRIB_PR_TEMPLATE.md)
Else, if not, please remove this block of text.
diff --git a/.github/PULL_REQUEST_TEMPLATE/HAMILTON_CONTRIB_PR_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE/HAMILTON_CONTRIB_PR_TEMPLATE.md
index 3a29aadd..6c714325 100644
--- a/.github/PULL_REQUEST_TEMPLATE/HAMILTON_CONTRIB_PR_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE/HAMILTON_CONTRIB_PR_TEMPLATE.md
@@ -10,8 +10,8 @@ Do you have the following?
- [ ] Added an __init__.py file under my username directory.
- [ ] Added a new folder for my dataflow under my username directory.
- [ ] Added a README.md file under my dataflow directory that follows the standard headings and is filled out.
- - [ ] Added a __init__.py file under my dataflow directory that contains the Hamilton code.
- - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Hamilton.
+ - [ ] Added a __init__.py file under my dataflow directory that contains the Apache Hamilton code.
+ - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Apache Hamilton.
- [ ] Added tags.json under my dataflow directory to curate my dataflow.
- [ ] Added valid_configs.jsonl under my dataflow directory to specify the valid configurations.
- [ ] Added a dag.png that shows one possible configuration of my dataflow.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index b515b649..800a4470 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -29,7 +29,7 @@ own fork, and then submit a pull request. All new code should have associated
unit tests that validate implemented features and the presence or lack of defects.
Additionally, the code should follow any stylistic and architectural guidelines
prescribed by the project. For us here, this means you install a pre-commit hook and use
-the given style files. Basically, you should mimic the styles and patterns in the Hamilton code-base.
+the given style files. Basically, you should mimic the styles and patterns in the Apache Hamilton code-base.
In terms of getting setup to develop, we invite you to read our [developer setup guide](developer_setup.md).
diff --git a/README.md b/README.md
index 1d16c86e..5012c8f3 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
<div align="center">
- <h1><img src="https://github.com/apache/hamilton/assets/2328071/feb6abaa-b6d5-4271-a320-0ae4a18d8aa7" width="50"/> Hamilton β portable & expressive <br> data transformation DAGs</h1>
+ <h1><img src="https://github.com/apache/hamilton/assets/2328071/feb6abaa-b6d5-4271-a320-0ae4a18d8aa7" width="50"/> Apache Hamilton β portable & expressive <br> data transformation DAGs</h1>
<a href='https://hamilton.dagworks.io/en/latest/?badge=latest'>
<img src='https://readthedocs.org/projects/hamilton/badge/?version=latest' alt='Documentation Status' />
</a><a href="https://www.python.org/downloads/" target="_blank">
@@ -19,7 +19,7 @@
<img src="https://img.shields.io/badge/DAGWorks-Follow-purple.svg?logo=linkedin" />
</a>
<a href="https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g" target="_blank">
- <img src="https://img.shields.io/badge/Hamilton-Join-purple.svg?logo=slack" alt="Hamilton Slack"/>
+ <img src="https://img.shields.io/badge/Apache Hamilton-Join-purple.svg?logo=slack" alt="Apache Hamilton Slack"/>
</a>
<a href="https://twitter.com/hamilton_os" target="_blank">
<img src="https://img.shields.io/badge/HamiltonOS-Follow-purple.svg?logo=X"/>
@@ -30,9 +30,9 @@
</div>
<br></br>
-Hamilton is a lightweight Python library for directed acyclic graphs (DAGs) of data transformations. Your DAG is **portable**; it runs anywhere Python runs, whether it's a script, notebook, Airflow pipeline, FastAPI server, etc. Your DAG is **expressive**; Hamilton has extensive features to define and modify the execution of a DAG (e.g., data validation, experiment tracking, remote execution).
+Apache Hamilton is a lightweight Python library for directed acyclic graphs (DAGs) of data transformations. Your DAG is **portable**; it runs anywhere Python runs, whether it's a script, notebook, Airflow pipeline, FastAPI server, etc. Your DAG is **expressive**; Apache Hamilton has extensive features to define and modify the execution of a DAG (e.g., data validation, experiment tracking, remote execution).
-To create a DAG, write regular Python functions that specify their dependencies with their parameters. As shown below, it results in readable code that can always be visualized. Hamilton loads that definition and automatically builds the DAG for you!
+To create a DAG, write regular Python functions that specify their dependencies with their parameters. As shown below, it results in readable code that can always be visualized. Apache Hamilton loads that definition and automatically builds the DAG for you!
<div align="center">
<img src="./docs/_static/abc_highlight.png" alt="Create a project" width="65%"/>
@@ -42,53 +42,53 @@ To create a DAG, write regular Python functions that specify their dependencies
</div>
<br>
-Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems, BI dashboards, and the [Hamilton UI](https://hamilton.dagworks.io/en/latest/concepts/ui) allows you to automatically visualize, catalog, and monitor execution.
+Apache Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems, BI dashboards, and the [Apache Hamilton UI](https://hamilton.dagworks.io/en/latest/concepts/ui) allows you to automatically visualize, catalog, and monitor execution.
-> Hamilton is great for DAGs, but if you need loops or conditional logic to create an LLM agent or a simulation, take a look at our sister library [Burr](https://github.com/apache/burr) π€ .
+> Apache Hamilton is great for DAGs, but if you need loops or conditional logic to create an LLM agent or a simulation, take a look at our sister library [Burr](https://github.com/apache/burr) π€ .
# Installation
-Hamilton supports Python 3.8+. We include the optional `visualization` dependency to display our Hamilton DAG. For visualizations, [Graphviz](https://graphviz.org/download/) needs to be installed on your system separately.
+Apache Hamilton supports Python 3.8+. We include the optional `visualization` dependency to display our Apache Hamilton DAG. For visualizations, [Graphviz](https://graphviz.org/download/) needs to be installed on your system separately.
```bash
pip install "sf-hamilton[visualization]"
```
-To use the Hamilton UI, install the `ui` and `sdk` dependencies.
+To use the Apache Hamilton UI, install the `ui` and `sdk` dependencies.
```bash
pip install "sf-hamilton[ui,sdk]"
```
-To try Hamilton in the browser, visit [www.tryhamilton.dev](https://www.tryhamilton.dev/?utm_source=README)
+To try Apache Hamilton in the browser, visit [www.tryhamilton.dev](https://www.tryhamilton.dev/?utm_source=README)
-# Why use Hamilton?
+# Why use Apache Hamilton?
-Data teams write code to deliver business value, but few have the resources to standardize practices and provide quality assurance. Moving from proof-of-concept to production and cross-function collaboration (e.g., data science, engineering, ops) remain challenging for teams, big or small. Hamilton is designed to help throughout a project's lifecycle:
+Data teams write code to deliver business value, but few have the resources to standardize practices and provide quality assurance. Moving from proof-of-concept to production and cross-function collaboration (e.g., data science, engineering, ops) remain challenging for teams, big or small. Apache Hamilton is designed to help throughout a project's lifecycle:
-- **Separation of concerns**. Hamilton separates the DAG "definition" and "execution" which lets data scientists focus on solving problems and engineers manage production pipelines.
+- **Separation of concerns**. Apache Hamilton separates the DAG "definition" and "execution" which lets data scientists focus on solving problems and engineers manage production pipelines.
-- **Effective collaboration**. The [Hamilton UI provides a shared interface](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/) for teams to inspect results and debug failures throughout the development cycle.
+- **Effective collaboration**. The [Apache Hamilton UI provides a shared interface](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/) for teams to inspect results and debug failures throughout the development cycle.
- **Low-friction dev to prod**. Use `@config.when()` to modify your DAG between execution environments instead of error-prone `if/else` feature flags. The notebook extension prevents the pain of migrating code from a notebook to a Python module.
- **Portable transformations**. Your DAG is [independent of infrastructure or orchestration](https://blog.dagworks.io/publish/posts/detail/145543927?referrer=%2Fpublish%2Fposts), meaning you can develop and debug locally and reuse code across contexts (local, Airflow, FastAPI, etc.).
-- **Maintainable DAG definition**. Hamilton [automatically builds the DAG from a single line of code whether it has 10 or 1000 nodes](https://hamilton.dagworks.io/en/latest/concepts/driver/). It can also assemble multiple Python modules into a pipeline, encouraging modularity.
+- **Maintainable DAG definition**. Apache Hamilton [automatically builds the DAG from a single line of code whether it has 10 or 1000 nodes](https://hamilton.dagworks.io/en/latest/concepts/driver/). It can also assemble multiple Python modules into a pipeline, encouraging modularity.
- **Expressive DAGs**. [Function modifiers](https://hamilton.dagworks.io/en/latest/concepts/function-modifiers/) are a unique feature to keep your code [DRY](https://en.wikipedia.org/wiki/Don't_repeat_yourself) and reduce the complexity of maintaining large DAGs. Other frameworks inevitably lead to code redundancy or bloated functions.
-- **Built-in coding style**. The Hamilton DAG is [defined using Python functions](https://hamilton.dagworks.io/en/latest/concepts/node/), encouraging modular, easy-to-read, self-documenting, and unit testable code.
+- **Built-in coding style**. The Apache Hamilton DAG is [defined using Python functions](https://hamilton.dagworks.io/en/latest/concepts/node/), encouraging modular, easy-to-read, self-documenting, and unit testable code.
- **Data and schema validation**. Decorate functions with `@check_output` to validate output properties, and raise warnings or exceptions. Add the `SchemaValidator()` adapter to automatically inspect dataframe-like objects (pandas, polars, Ibis, etc.) to track and validate their schema.
-- **Built for plugins**. Hamilton is designed to play nice with all tools and provides the right abstractions to create custom integrations with your stack. Our lively community will help you build what you need!
+- **Built for plugins**. Apache Hamilton is designed to play nice with all tools and provides the right abstractions to create custom integrations with your stack. Our lively community will help you build what you need!
-# Hamilton UI
+# Apache Hamilton UI
-You can track the execution of your Hamilton DAG in the [Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/). It automatically populates a data catalog with lineage / tracing and provides execution observability to inspect results and debug errors. You can run it as a [local server](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/#local-mode) or a [self-hosted application using Docker](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/#docker-deployed-mode).
+You can track the execution of your Apache Hamilton DAG in the [Apache Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/). It automatically populates a data catalog with lineage / tracing and provides execution observability to inspect results and debug errors. You can run it as a [local server](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/#local-mode) or a [self-hosted application using Docker](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/#docker-depl [...]
<p align="center">
<img src="./docs/_static/hamilton_1.jpeg" alt="Description1" width="30%" style="margin-right: 20px;"/>
@@ -100,9 +100,9 @@ You can track the execution of your Hamilton DAG in the [Hamilton UI](https://ha
</p>
-## Get started with the Hamilton UI
+## Get started with the Apache Hamilton UI
-1. To use the Hamilton UI, install the dependencies (see `Installation` section) and start the server with
+1. To use the Apache Hamilton UI, install the dependencies (see `Installation` section) and start the server with
```bash
hamilton ui
@@ -115,7 +115,7 @@ You can track the execution of your Hamilton DAG in the [Hamilton UI](https://ha
</div>
<br>
-3. Track your Hamilton DAG by creating a `HamiltonTracker` object with your `username` and `project_id` and adding it to your `Builder`. Now, your DAG will appear in the UI's catalog and all executions will be tracked!
+3. Track your Apache Hamilton DAG by creating a `HamiltonTracker` object with your `username` and `project_id` and adding it to your `Builder`. Now, your DAG will appear in the UI's catalog and all executions will be tracked!
```python
from hamilton import driver
@@ -143,7 +143,7 @@ You can track the execution of your Hamilton DAG in the [Hamilton UI](https://ha
# Documentation & learning resources
-* π See the [official documentation](https://hamilton.dagworks.io/) to learn about the core concepts of Hamilton.
+* π See the [official documentation](https://hamilton.dagworks.io/) to learn about the core concepts of Apache Hamilton.
* π¨βπ« Consult the [examples on GitHub](https://github.com/apache/hamilton/tree/main/examples) to learn about specific features or integrations with other frameworks.
@@ -151,14 +151,14 @@ You can track the execution of your Hamilton DAG in the [Hamilton UI](https://ha
* πΊ Find video tutorials on the [DAGWorks YouTube channel](https://www.youtube.com/@DAGWorks-Inc)
-* π£ Reach out via the [Hamilton Slack community](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g) for help and troubleshooting
+* π£ Reach out via the [Apache Hamilton Slack community](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g) for help and troubleshooting
-# How does Hamilton compare to X?
+# How does Apache Hamilton compare to X?
-Hamilton is not an orchestrator ([you might not need one](https://blog.dagworks.io/p/lean-data-automation-a-principal)), nor a feature store ([but you can use it to build one!](https://blog.dagworks.io/p/featurization-integrating-hamilton)). Its purpose is to help you structure and manage data transformations. If you know dbt, Hamilton does for Python what dbt does for SQL.
+Apache Hamilton is not an orchestrator ([you might not need one](https://blog.dagworks.io/p/lean-data-automation-a-principal)), nor a feature store ([but you can use it to build one!](https://blog.dagworks.io/p/featurization-integrating-hamilton)). Its purpose is to help you structure and manage data transformations. If you know dbt, Apache Hamilton does for Python what dbt does for SQL.
-Another way to frame it is to think about the different layers of a data stack. Hamilton is at the **asset layer**. It helps you organize data transformations code (the **expression layer**), manage changes, and validate & test data.
+Another way to frame it is to think about the different layers of a data stack. Apache Hamilton is at the **asset layer**. It helps you organize data transformations code (the **expression layer**), manage changes, and validate & test data.
<div align="center" style="width: 100%">
<table>
@@ -183,7 +183,7 @@ Another way to frame it is to think about the different layers of a data stack.
<tr>
<td>Asset</td>
<td>Organize expressions into meaningful units <br> (e.g., dataset, ML model, table)</td>
- <td><b>Hamilton</b>, dbt, dlt, SQLMesh, <a href="https://github.com/apache/burr">Burr</a></td>
+ <td><b>Apache Hamilton</b>, dbt, dlt, SQLMesh, <a href="https://github.com/apache/burr">Burr</a></td>
</tr>
<tr>
<td>Expression</td>
@@ -204,11 +204,11 @@ Another way to frame it is to think about the different layers of a data stack.
</table>
</div>
-See our page on [Why use Hamilton?](https://hamilton.dagworks.io/en/latest/get-started/why-hamilton/) and framework [code comparisons](https://hamilton.dagworks.io/en/latest/code-comparisons/) for more information.
+See our page on [Why use Apache Hamilton?](https://hamilton.dagworks.io/en/latest/get-started/why-hamilton/) and framework [code comparisons](https://hamilton.dagworks.io/en/latest/code-comparisons/) for more information.
# π License
-Hamilton is released under the Apache 2.0 License. See [LICENSE](https://github.com/apache/hamilton/blob/main/LICENSE.md) for details.
+Apache Hamilton is released under the Apache 2.0 License. See [LICENSE](https://github.com/apache/hamilton/blob/main/LICENSE.md) for details.
# π Community
@@ -219,7 +219,7 @@ We're very supportive of changes by new contributors, big or small! Make sure to
## π Used by
-Hamilton was started at Stitch Fix before the original creators founded DAGWorks Inc! The library is battle-tested and has been supporting production use cases since 2019.
+Apache Hamilton was started at Stitch Fix before the original creators founded DAGWorks Inc! The library is battle-tested and has been supporting production use cases since 2019.
>Read more about the [origin story](https://multithreaded.stitchfix.com/blog/2021/10/14/functions-dags-hamilton/).
@@ -260,13 +260,13 @@ Hamilton was started at Stitch Fix before the original creators founded DAGWorks
## π Special Mentions & π¦ Bug Hunters
-Thanks to our awesome community and their active involvement in the Hamilton library.
+Thanks to our awesome community and their active involvement in the Apache Hamilton library.
[Nils Olsson](https://github.com/nilsso), [MichaΕ Siedlaczek](https://github.com/elshize), [Alaa Abedrabbo](https://github.com/AAbedrabbo), [Shreya Datar](https://github.com/datarshreya), [Baldo Faieta](https://github.com/baldofaieta), [Anwar Brini](https://github.com/AnwarBrini), [Gourav Kumar](https://github.com/gms101), [Amos Aikman](https://github.com/amosaikman), [Ankush Kundaliya](https://github.com/akundaliya), [David Weselowski](https://github.com/j7zAhU), [Peter Robinson](https: [...]
Kemal Eren](https://github.com/kemaleren), [Jernej Frank](https://github.com/jernejfrank), [Noah Ridge](https://github.com/noahridge)
# π Citations
-We'd appreciate citing Hamilton by referencing one of the following:
+We'd appreciate citing Apache Hamilton by referencing one of the following:
```bibtex
@inproceedings{DBLP:conf/vldb/KrawczykI22,
@@ -295,9 +295,9 @@ We'd appreciate citing Hamilton by referencing one of the following:
}
```
-# π Libraries built on / for Hamilton
+# π Libraries built on / for Apache Hamilton
* [Hypster](https://github.com/gilad-rubin/hypster) - hyperparameter management
* [DSP Decision Engine](https://github.com/capitec/dsp-decision-engine) - decision trees
* [NaturF](https://github.com/IMMM-SFA/naturf/) - library for data transformation for weather forecasting
* [WrenAI](https://github.com/Canner/WrenAI) - RAG
-* [FlowerPower](https://github.com/legout/flowerpower/) - Scheduler for Hamilton
+* [FlowerPower](https://github.com/legout/flowerpower/) - Scheduler for Apache Hamilton
diff --git a/basics.md b/basics.md
index c0bc6dd4..e848ea59 100644
--- a/basics.md
+++ b/basics.md
@@ -1,19 +1,19 @@
-# Hamilton Basics
+# Apache Hamilton Basics
-There are two parts to Hamilton:
+There are two parts to Apache Hamilton:
-1. Hamilton Functions.
+1. Apache Hamilton Functions.
- Hamilton Functions are what you, the end user write.
+ Apache Hamilton Functions are what you, the end user write.
-2. Hamilton Driver.
+2. Apache Hamilton Driver.
- Once you've written your functions, you will need to use the Hamilton Driver to build the DAG and orchestrate
+ Once you've written your functions, you will need to use the Apache Hamilton Driver to build the DAG and orchestrate
execution.
Let's dive deeper into these parts below, but first a word on terminology.
-We use the following terms interchangeably, e.g. a ____ in Hamilton is ... :
+We use the following terms interchangeably, e.g. a ____ in Apache Hamilton is ... :
* column
* variable
@@ -24,10 +24,10 @@ That's because we're representing columns as functions, which are parts of a dir
a column is a part of a dataframe. To compute a column we write a function that has input variables. From these functions
we create a DAG and represent each function as a node, linking each input variable by an edge to its respective node.
-## Hamilton Functions
-Using Hamilton is all about writing functions. From these functions a dataframe is constructed for you at execution time.
+## Apache Hamilton Functions
+Using Apache Hamilton is all about writing functions. From these functions a dataframe is constructed for you at execution time.
-A simple (but rather contrived) example of what Hamilton does that adds two numbers is as follows:
+A simple (but rather contrived) example of what Apache Hamilton does that adds two numbers is as follows:
```python
def _sum(*vars):
@@ -51,20 +51,20 @@ While this looks like a simple python function, there are a few components to no
and separate out that logic for debugging/iterating.
2. The function `sum_a_b` depends on two upstream nodes -- `a` and `b`. This means that these values must either be:
* Defined by another function
- * Passed in by the user as a configuration variable (see `Hamilton Driver Code` below)
-3. The function `sum_a_b` makes full use of the python type-hint system. This is required in Hamilton,
+ * Passed in by the user as a configuration variable (see `Apache Hamilton Driver Code` below)
+3. The function `sum_a_b` makes full use of the python type-hint system. This is required in Apache Hamilton,
as it allows us to type-check the inputs and outputs to match with upstream producers and downstream consumers. In this case,
we know that the input `a` has to be an integer, the input `b` has to also be an integer, and anything that declares `sum_a_b` as an input
has to declare it as an integer.
4. Standard python documentation is a first-class citizen. As we have a 1:1 relationship between python functions and
nodes, each function documentation also describes a piece of business logic.
-5. Functions that start with _ are ignored, and not included in the DAG. Hamilton tries to make use of every function
+5. Functions that start with _ are ignored, and not included in the DAG. Apache Hamilton tries to make use of every function
in a module, so this allows us to easily indicate helper functions that won't become part of the DAG.
-### Python Types & Hamilton
+### Python Types & Apache Hamilton
-Hamilton makes use of python's type-hinting feature to check compatibility between function outputs and function inputs. However,
+Apache Hamilton makes use of python's type-hinting feature to check compatibility between function outputs and function inputs. However,
this is not particularly sophisticated, largely due to the lack of available tooling in python. Thus, generic types do not function correctly.
The following will not work:
@@ -86,8 +86,8 @@ def some_func() -> dict:
While this is unfortunate, the typing API in python is not yet sophisticated enough to rely on accurate subclass validation.
-## Hamilton Driver Code
-For documentation on the actual Hamilton Driver code, we invite the reader to [read the Driver class source code](/hamilton/driver.py) directly.
+## Apache Hamilton Driver Code
+For documentation on the actual Apache Hamilton Driver code, we invite the reader to [read the Driver class source code](/hamilton/driver.py) directly.
At a high level, the driver code does two things:
@@ -115,7 +115,7 @@ The configuration is used not just to feed data to the DAG, but also to determin
As such, it is passed in to the constructor, and used during DAG creation. This enables such decorators like @config.when.
Otherwise the contents of the _config_ dictionary should include all the inputs required for whatever final output you
-want to create. The configuration dictionary should not be used for overriding what Hamilton will compute.
+want to create. The configuration dictionary should not be used for overriding what Apache Hamilton will compute.
To do this, use the `override` parameter as part of the `execute()` -- see below.
#### \*modules: ModuleType
@@ -130,13 +130,13 @@ via a recursive depth-first-traversal, which leads to the possibility (although
recursion depth errors. If that happens, the culprit is almost always a circular reference in the graph. We suggest
displaying the DAG to verify this.
-To help speed up development of new or existing Hamilton Functions, we enable you to _override_ parts of the DAG. What
-this means is that before calling `execute()`, you have computed some result that you want to use instead of what Hamilton
+To help speed up development of new or existing Apache Hamilton Functions, we enable you to _override_ parts of the DAG. What
+this means is that before calling `execute()`, you have computed some result that you want to use instead of what Apache Hamilton
would produce. To do so, you just pass in a dictionary of `{'col_name': YOUR_VALUE}` as the overrides argument to the
execute function.
To visualize the DAG that would be executed, pass the flag `display_graph=True` to execute. It will render an image in a pdf format.
# Backstory
-For the backstory on Hamilton we invite you to watch ~9 minute lightning talk on it that we gave at the apply conference:
+For the backstory on Apache Hamilton we invite you to watch ~9 minute lightning talk on it that we gave at the apply conference:
[video](https://www.youtube.com/watch?v=B5Zp_30Knoo), [slides](https://www.slideshare.net/StefanKrawczyk/hamilton-a-micro-framework-for-creating-dataframes).
diff --git a/contrib/README.md b/contrib/README.md
index 8b82eab5..17803192 100644
--- a/contrib/README.md
+++ b/contrib/README.md
@@ -1,12 +1,12 @@
-## Off-the-shelf Hamilton Dataflows
+## Off-the-shelf Apache Hamilton Dataflows
-Here you'll find documentation about the sf-hamilton-contrib package that curates a collection of Hamilton Dataflows that are
+Here you'll find documentation about the sf-hamilton-contrib package that curates a collection of Apache Hamilton Dataflows that are
ready to be used in your own projects. They are user-contributed and maintained, with
-the goal of making it easier for you to get started with Hamilton.
+the goal of making it easier for you to get started with Apache Hamilton.
We expect this collection to grow over time, so check back often! As dataflows become mature we
will move them into the official sub-package of the respository and become maintained by the
-Hamilton team.
+Apache Hamilton team.
### Usage
There are two ways to get access to dataflows in this package. For either approach,
@@ -132,8 +132,8 @@ This will do the following:
- `__init__.py` -- this will be an empty file that allows you to import your dataflow
2. Create a package under `contrib/hamilton/contrib/user/<your_github_username>/<name_of_dataflow>` with the appropriate files to describe your dataflow:
- `README.md` to describe the dataflow with the standard headings
- - `__init__.py` to contain the Hamilton code
- - `requirements.txt` to contain the required packages outside of Hamilton
+ - `__init__.py` to contain the Apache Hamilton code
+ - `requirements.txt` to contain the required packages outside of Apache Hamilton
- `tags.json` to curate your dataflow
- `valid_configs.jsonl` to specify the valid configurations for it to be run
- `dag.png` to show one possible configuration of your dataflow
@@ -165,7 +165,7 @@ dr = driver.Driver(
dr.display_all_functions("dag", {"format": "png", "view": False})
```
- [ ] Push a branch back to your fork
-- [ ] Open up a pull request to the main Hamilton repo!
+- [ ] Open up a pull request to the main Apache Hamilton repo!
- [ ] Commit the files we just added
- [ ] Create a PR
- [ ] Tag one of the maintainers [elijahbenizzy](https://github.com/elijahbenizzy), [skrawcz](https://github.com/skrawcz), or [zilto](https://github.com/zilto) for a review
diff --git a/contrib/docs/blog/2023-11-13-welcome/index.md b/contrib/docs/blog/2023-11-13-welcome/index.md
index 150fa997..dc31e7cf 100644
--- a/contrib/docs/blog/2023-11-13-welcome/index.md
+++ b/contrib/docs/blog/2023-11-13-welcome/index.md
@@ -1,14 +1,14 @@
---
slug: hub-launch-welcome
-title: π Welcome to the Hamilton Hub
+title: π Welcome to the Apache Hamilton Hub
authors: [skrawcz]
tags: [hamilton, how-to, dataflows]
---
-Welcome to the Hamilton Dataflow Hub!
+Welcome to the Apache Hamilton Dataflow Hub!
We're excited to kick off this project and hope that it will be a useful resource for the community.
-Hamilton is like a Swiss army knife, it's very versatile, as you can use it for data, ML, LLM, and web work!
+Apache Hamilton is like a Swiss army knife, it's very versatile, as you can use it for data, ML, LLM, and web work!
We envision this hub, as a place to share the flows you've built that can be useful to others! As well as a place
for you to find flows that others have built that you can use and easily modify for your work!
@@ -22,7 +22,7 @@ for you to find flows that others have built that you can use and easily modify
## How to contribute
-We're excited to have you contribute to the Hamilton Dataflow Hub!
+We're excited to have you contribute to the Apache Hamilton Dataflow Hub!
Head on over to [github and read the instructions](https://github.com/apache/hamilton/tree/main/contrib#how-to-contribute).
diff --git a/contrib/docs/docs/README.md b/contrib/docs/docs/README.md
index a8046741..397c3276 100644
--- a/contrib/docs/docs/README.md
+++ b/contrib/docs/docs/README.md
@@ -3,17 +3,17 @@ title: "Introduction"
sidebar_position: 1
---
-# Hamilton Dataflows
+# Apache Hamilton Dataflows
<h3> Welcome!</h3>
-Here you'll find a website that curates a collection of Hamilton Dataflows that are
+Here you'll find a website that curates a collection of Apache Hamilton Dataflows that are
ready to be used in your own projects. They are user-contributed and maintained, with
-the goal of making it easier for you to get started with Hamilton.
+the goal of making it easier for you to get started with Apache Hamilton.
We expect this collection to grow over time, so check back often! As dataflows become mature we
will move them into the official DAGWorks sub-package of this site and become maintained by
-DAGWorks's Hamilton team.
+DAGWorks's Apache Hamilton team.
## Navigation
βοΈ Use the search bar above to quickly find dataflows based on keyword search.
@@ -26,13 +26,13 @@ There are two methods to get access to dataflows presented here.
Assumptions:
-1. You are familiar with Hamilton and have it installed. If not, take
-[15 minutes to learn Hamilton in your browser](https://www.tryhamilton.dev/) and then `pip install sf-hamilton` to get started.
-Come back here when you're ready to use Hamilton.
+1. You are familiar with Apache Hamilton and have it installed. If not, take
+[15 minutes to learn Apache Hamilton in your browser](https://www.tryhamilton.dev/) and then `pip install sf-hamilton` to get started.
+Come back here when you're ready to use Apache Hamilton.
2. You have the requisite python dependencies installed on your system.
You'll get import errors if you don't. Don't know what you need? Scroll to the bottom of a dataflow to find the requirements. We're working on convenience functions to help!
-For more extensive documentation, please see [Hamilton User Contrib documentation](https://hamilton.dagworks.io).
+For more extensive documentation, please see [Apache Hamilton User Contrib documentation](https://hamilton.dagworks.io).
### Dynamic installation
Here we dynamically download the dataflow from the internet and execute it. This is useful for quickly
@@ -138,8 +138,8 @@ Do you have the following?
- [ ] Added an __init__.py file under my username directory.
- [ ] Added a new folder for my dataflow under my username directory.
- [ ] Added a README.md file under my dataflow directory that follows the standard headings and is filled out.
- - [ ] Added a __init__.py file under my dataflow directory that contains the Hamilton code.
- - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Hamilton.
+ - [ ] Added a __init__.py file under my dataflow directory that contains the Apache Hamilton code.
+ - [ ] Added a requirements.txt under my dataflow directory that contains the required packages outside of Apache Hamilton.
- [ ] Added tags.json under my dataflow directory to curate my dataflow.
- [ ] Added valid_configs.jsonl under my dataflow directory to specify the valid configurations.
- [ ] Added a dag.png that shows one possible configuration of my dataflow.
diff --git a/contrib/hamilton/contrib/dagworks/author.md b/contrib/hamilton/contrib/dagworks/author.md
index c9577e8c..579438bd 100644
--- a/contrib/hamilton/contrib/dagworks/author.md
+++ b/contrib/hamilton/contrib/dagworks/author.md
@@ -1,7 +1,7 @@
---
title: DAGWorks
---
-# Officially maintained Hamilton dataflows
+# Officially maintained Apache Hamilton dataflows
Here you'll find dataflows that are officially maintained by the DAGWorks team. Popular
user-contributed dataflows will be moved here as necessary to support their continued
diff --git a/contrib/hamilton/contrib/dagworks/conversational_rag/README.md b/contrib/hamilton/contrib/dagworks/conversational_rag/README.md
index 4d5593eb..67697036 100644
--- a/contrib/hamilton/contrib/dagworks/conversational_rag/README.md
+++ b/contrib/hamilton/contrib/dagworks/conversational_rag/README.md
@@ -1,7 +1,7 @@
# Purpose of this module
This module shows a conversational retrieval augmented generation (RAG) example using
-Hamilton. It shows you how you might structure your code with Hamilton to
+Apache Hamilton. It shows you how you might structure your code with Apache Hamilton to
create a RAG pipeline that takes into account conversation.
This example uses [FAISS](https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/) + and in memory vector store and the OpenAI LLM provider.
@@ -25,7 +25,7 @@ These are the defined inputs you can provide.
- *top_k*: An integer. This is the number of vectors to retrieve from the vector store. Defaults to 5.
### Overrides
-With Hamilton you can easily override a function and provide a value for it. For example if you're
+With Apache Hamilton you can easily override a function and provide a value for it. For example if you're
iterating you might just want to override these two values before modifying the functions:
- *context*: if you want to skip going to the vector store and provide the context directly, you can do so by providing this override.
@@ -91,7 +91,7 @@ What you'd most likely want to do is:
3. Change the context and prompt.
With (1) you can import any vector store/library that you want. You should draw out
-the process you would like, and that should then map to Hamilton functions.
+the process you would like, and that should then map to Apache Hamilton functions.
With (2) you can import any LLM provider that you want, just use `@config.when` if you
want to switch between multiple providers.
With (3) you can add more functions that create parts of the prompt.
diff --git a/contrib/hamilton/contrib/dagworks/faiss_rag/README.md b/contrib/hamilton/contrib/dagworks/faiss_rag/README.md
index 2b6c8325..9f13f936 100644
--- a/contrib/hamilton/contrib/dagworks/faiss_rag/README.md
+++ b/contrib/hamilton/contrib/dagworks/faiss_rag/README.md
@@ -1,7 +1,7 @@
# Purpose of this module
This module shows a simple retrieval augmented generation (RAG) example using
-Hamilton. It shows you how you might structure your code with Hamilton to
+Apache Hamilton. It shows you how you might structure your code with Apache Hamilton to
create a simple RAG pipeline.
This example uses [FAISS](https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/) + and in memory vector store and the OpenAI LLM provider.
@@ -19,7 +19,7 @@ These are the defined inputs.
- *top_k*: An integer. This is the number of vectors to retrieve from the vector store. Defaults to 5.
### Overrides
-With Hamilton you can easily override a function and provide a value for it. For example if you're
+With Apache Hamilton you can easily override a function and provide a value for it. For example if you're
iterating you might just want to override these two values before modifying the functions:
- *context*: if you want to skip going to the vector store and provide the context directly, you can do so by providing this override.
@@ -63,7 +63,7 @@ What you'd most likely want to do is:
3. Change the context and prompt.
With (1) you can import any vector store/library that you want. You should draw out
-the process you would like, and that should then map to Hamilton functions.
+the process you would like, and that should then map to Apache Hamilton functions.
With (2) you can import any LLM provider that you want, just use `@config.when` if you
want to switch between multiple providers.
With (3) you can add more functions that create parts of the prompt.
diff --git a/contrib/hamilton/contrib/dagworks/simple_eval_grader/README.md b/contrib/hamilton/contrib/dagworks/simple_eval_grader/README.md
index 0aca3b01..21bea413 100644
--- a/contrib/hamilton/contrib/dagworks/simple_eval_grader/README.md
+++ b/contrib/hamilton/contrib/dagworks/simple_eval_grader/README.md
@@ -12,7 +12,7 @@ These are the defined inputs you can provide.
- *question*: The user question that specifies what to generate for the quiz
### Overrides
-With Hamilton you can easily override a function and provide a value for it. For example if you're
+With Apache Hamilton you can easily override a function and provide a value for it. For example if you're
iterating you might just want to override these two values before modifying the functions:
- *quiz_bank*: provide a fixed string of context that is the facts for the quiz bank.
diff --git a/contrib/hamilton/contrib/dagworks/translate_to_hamilton/README.md b/contrib/hamilton/contrib/dagworks/translate_to_hamilton/README.md
index d76508b7..3881c96d 100644
--- a/contrib/hamilton/contrib/dagworks/translate_to_hamilton/README.md
+++ b/contrib/hamilton/contrib/dagworks/translate_to_hamilton/README.md
@@ -1,7 +1,7 @@
# Purpose of this module
The purpose of this module is to provide you with a dataflow to translate
-your existing procedural code into code written in the Hamilton style.
+your existing procedural code into code written in the Apache Hamilton style.
```python
# Make sure you have an API key in your environment, e.g. os.environ["OPENAI_API_KEY"]
diff --git a/contrib/hamilton/contrib/user/elijahbenizzy/author.md b/contrib/hamilton/contrib/user/elijahbenizzy/author.md
index 4a039217..7cc615c4 100644
--- a/contrib/hamilton/contrib/user/elijahbenizzy/author.md
+++ b/contrib/hamilton/contrib/user/elijahbenizzy/author.md
@@ -1,6 +1,6 @@
# elijahbenizzy
-Elijah is one of the co-authors of Hamilton! He loves building out tooling for clean, reliable, and scalable dataflows.
+Elijah is one of the co-authors of Apache Hamilton! He loves building out tooling for clean, reliable, and scalable dataflows.
In his spare time, he enjoys cycling, cooking, reading antique maps, learning about contemporary history, and hacking with fractals.
diff --git a/contrib/hamilton/contrib/user/skrawcz/author.md b/contrib/hamilton/contrib/user/skrawcz/author.md
index 273c1a33..0fcb5eee 100644
--- a/contrib/hamilton/contrib/user/skrawcz/author.md
+++ b/contrib/hamilton/contrib/user/skrawcz/author.md
@@ -1,6 +1,6 @@
# skrawcz
-Hi I'm Stefan Krawczyk. I am one of the co-creators of Hamilton.
+Hi I'm Stefan Krawczyk. I am one of the co-creators of Apache Hamilton.
# Github
https://github.com/skrawcz
diff --git a/contrib/hamilton/contrib/user/skrawcz/customize_embeddings/README.md b/contrib/hamilton/contrib/user/skrawcz/customize_embeddings/README.md
index 74e25770..59bad8d8 100644
--- a/contrib/hamilton/contrib/user/skrawcz/customize_embeddings/README.md
+++ b/contrib/hamilton/contrib/user/skrawcz/customize_embeddings/README.md
@@ -76,5 +76,5 @@ The module uses libraries such as numpy, pandas, plotly, torch, sklearn, and ope
# Limitations
This code is currently set up to work with OpenAI. It could be modified to work with other embedding producers/providers.
-The matrix optimization is not set up to be parallelized, though it could be done using Hamilton constructs,
+The matrix optimization is not set up to be parallelized, though it could be done using Apache Hamilton constructs,
like `Parallelizeable`.
diff --git a/contrib/hamilton/contrib/user/skrawcz/fine_tuning/README.md b/contrib/hamilton/contrib/user/skrawcz/fine_tuning/README.md
index 9bb8204f..854f9223 100644
--- a/contrib/hamilton/contrib/user/skrawcz/fine_tuning/README.md
+++ b/contrib/hamilton/contrib/user/skrawcz/fine_tuning/README.md
@@ -52,7 +52,7 @@ e.g. you should be able to do `json.load(f)` and it would return a list of dicti
"reply": "42"
},
{
- "question": "What is Hamilton?",
+ "question": "What is Apache Hamilton?",
"reply": "..."
},
...
diff --git a/contrib/hamilton/contrib/user/zilto/xgboost_optuna/README.md b/contrib/hamilton/contrib/user/zilto/xgboost_optuna/README.md
index 2d755611..f85ba2e5 100644
--- a/contrib/hamilton/contrib/user/zilto/xgboost_optuna/README.md
+++ b/contrib/hamilton/contrib/user/zilto/xgboost_optuna/README.md
@@ -5,7 +5,7 @@ This module implements a dataflow to train an XGBoost model with hyperparameter
You give it a 2D arrays for `X_train`, `y_train`, `X_test`, `y_test` and you are good to go!
# Configuration Options
-The Hamilton driver can be configured with the following options:
+The Apache Hamilton driver can be configured with the following options:
- {"task": "classification"} to use xgboost.XGBClassifier.
- {"task": "regression"} to use xgboost.XGBRegressor.
diff --git a/data_quality.md b/data_quality.md
index 75b3aa98..3c5ce9f2 100644
--- a/data_quality.md
+++ b/data_quality.md
@@ -1,6 +1,6 @@
# Data Quality
-Hamilton has a simple but powerful data quality capability. This enables you to write functions
+Apache Hamilton has a simple but powerful data quality capability. This enables you to write functions
that have assertion on the output. For example...
```python
diff --git a/decorators.md b/decorators.md
index e815fddb..89ebf462 100644
--- a/decorators.md
+++ b/decorators.md
@@ -68,7 +68,7 @@ tuple(Output Name, Documentation string) -> value.
Note that `@parametrized` is deprecated, and we intend for you to use `@parameterize_vales`. We're consolidating
to make the parameterization decorators more consistent! You have plenty of time to migrate,
-we wont make this a hard change until we have a Hamilton 2.0.0 to release.
+we wont make this a hard change until we have a Apache Hamilton 2.0.0 to release.
## @parameterize_sources (replacing @parameterized_inputs)
@@ -401,7 +401,7 @@ See [data_quality](data_quality.md) for more information on available validators
## @subdag
The `@subdag` decorator enables you to rerun components of your DAG with varying parameters. Note that this is immensely powerful -- if we
-draw analogies from Hamilton to standard procedural programming paradigms, we might have the following correspondence:
+draw analogies from Apache Hamilton to standard procedural programming paradigms, we might have the following correspondence:
- `config.when` + friends -- `if/else` statements
- `parameterize`/`extract_columns` -- `for` loop
diff --git a/dev_tools/language_server/README.md b/dev_tools/language_server/README.md
index b299a816..68396cd7 100644
--- a/dev_tools/language_server/README.md
+++ b/dev_tools/language_server/README.md
@@ -1,5 +1,5 @@
-# Hamilton Language Server (`hamilton_lsp`)
+# Apache Hamilton Language Server (`hamilton_lsp`)
-This is an implementation of the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) to provide a rich IDE experience when creating Hamilton dataflows.
+This is an implementation of the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) to provide a rich IDE experience when creating Apache Hamilton dataflows.
-It currently powers the Hamilton VSCode extension and could be integrated into other IDEs.
+It currently powers the Apache Hamilton VSCode extension and could be integrated into other IDEs.
diff --git a/dev_tools/vscode_extension/README.md b/dev_tools/vscode_extension/README.md
index 9259e173..8d75852e 100644
--- a/dev_tools/vscode_extension/README.md
+++ b/dev_tools/vscode_extension/README.md
@@ -1,4 +1,4 @@
-# Hamilton - VSCode extension
+# Apache Hamilton - VSCode extension
diff --git a/dev_tools/vscode_extension/resources/walkthrough/4_visitLSP.md b/dev_tools/vscode_extension/resources/walkthrough/4_visitLSP.md
index fe8d1489..0e0609a1 100644
--- a/dev_tools/vscode_extension/resources/walkthrough/4_visitLSP.md
+++ b/dev_tools/vscode_extension/resources/walkthrough/4_visitLSP.md
@@ -1,6 +1,6 @@
-# Hamilton Language Server
+# Apache Hamilton Language Server
-Hamilton VSCode uses a [Language Server](https://microsoft.github.io/language-server-protocol/) to build a dataflow from your currently active file.
+Apache Hamilton VSCode uses a [Language Server](https://microsoft.github.io/language-server-protocol/) to build a dataflow from your currently active file.
Current features include:
- view the dataflow
diff --git a/dev_tools/vscode_extension/resources/walkthrough/5_visitDataflowWebview.md b/dev_tools/vscode_extension/resources/walkthrough/5_visitDataflowWebview.md
index 37bea965..7af1584f 100644
--- a/dev_tools/vscode_extension/resources/walkthrough/5_visitDataflowWebview.md
+++ b/dev_tools/vscode_extension/resources/walkthrough/5_visitDataflowWebview.md
@@ -1,6 +1,6 @@
# Dataflow View
-The **dataflow webview** displays the Hamilton dataflow of the currently opened file.
+The **dataflow webview** displays the Apache Hamilton dataflow of the currently opened file.
## Toolbar features
- **Rotate**: rotates display by 90 degrees
diff --git a/dev_tools/vscode_extension/resources/walkthrough/6_tips.md b/dev_tools/vscode_extension/resources/walkthrough/6_tips.md
index 513de99a..ef4367e4 100644
--- a/dev_tools/vscode_extension/resources/walkthrough/6_tips.md
+++ b/dev_tools/vscode_extension/resources/walkthrough/6_tips.md
@@ -10,5 +10,5 @@ In VSCode, the vertical sidebar is called the `activity bar` (`ctrl+b`) and the
- The `activity bar` can be moved to the **left** or **right** via the setting `workbench.sideBar.location`
- The `panel` can be move to the **left**, **right**, or **bottom** by clicking the header and setting the `panel position`.
- The `panel` can be quickly open/closed with `ctrl+j`
-- If you can't find a view, open the command palet and search for `Hamilton: Focus [...]` for shortcuts.
+- If you can't find a view, open the command palet and search for `Apache Hamilton: Focus [...]` for shortcuts.
- If your setup becomes a mess, open the command palet and use `View: Reset View Locations`
diff --git a/developer_setup.md b/developer_setup.md
index 5048927c..11f6c67f 100644
--- a/developer_setup.md
+++ b/developer_setup.md
@@ -140,7 +140,7 @@ These are the steps to push to pypi. This is taken from the [python packaging tu
2. Setup API tokens and add them to your ~/.pypirc.
3. Run `python3 -m pip install --upgrade build`.
4. Run `python3 -m pip install --upgrade twine`
-5. Run `python3 -m build` to build Hamilton. It should create things in dist/*.
+5. Run `python3 -m build` to build Apache Hamilton. It should create things in dist/*.
6. Push to test pypi - `python3 -m twine upload --repository testpypi dist/*`.
Note: you cannot push the same version twice to test or real pypi. So it's useful to append `-rcX` to the version.
diff --git a/docs/code-comparisons/airflow.rst b/docs/code-comparisons/airflow.rst
index b6e80da3..fba602b2 100644
--- a/docs/code-comparisons/airflow.rst
+++ b/docs/code-comparisons/airflow.rst
@@ -2,39 +2,39 @@
Airflow
======================
-For more details see this `Hamilton + Airflow blog post <https://blog.dagworks.io/p/supercharge-your-airflow-dag-with>`_.
+For more details see this `Apache Hamilton + Airflow blog post <https://blog.dagworks.io/p/supercharge-your-airflow-dag-with>`_.
**TL;DR:**
-1. Hamilton complements Airflow. It'll help you write better, more modular, and testable code.
-2. Hamilton does not replace Airflow.
+1. Apache Hamilton complements Airflow. It'll help you write better, more modular, and testable code.
+2. Apache Hamilton does not replace Airflow.
High-level differences:
-----------------------
-* Hamilton is a micro-orchestator. Airflow is a macro-orchestrator.
-* Hamilton is a Python library standardizing how you express python pipelines, while Airflow is a complete platform and
+* Apache Hamilton is a micro-orchestator. Airflow is a macro-orchestrator.
+* Apache Hamilton is a Python library standardizing how you express python pipelines, while Airflow is a complete platform and
system for scheduling and executing pipelines.
-* Hamilton focuses on providing a lightweight, low dependency, flexible way to define data pipelines as Python functions,
+* Apache Hamilton focuses on providing a lightweight, low dependency, flexible way to define data pipelines as Python functions,
whereas Airflow is a whole system that comes with a web-based UI, scheduler, and executor.
-* Hamilton pipelines are defined using pure Python code, that can be run anywhere that Python runs. While Airflow uses
+* Apache Hamilton pipelines are defined using pure Python code, that can be run anywhere that Python runs. While Airflow uses
Python to describe a DAG, this DAG can only be run by the Airflow system.
-* Hamilton complements Airflow, and you can use Hamilton within Airflow. But the reverse is not true.
-* You can use Hamilton directly in a Jupyter Notebook, or Python web-service. You can't do this with Airflow.
+* Apache Hamilton complements Airflow, and you can use Apache Hamilton within Airflow. But the reverse is not true.
+* You can use Apache Hamilton directly in a Jupyter Notebook, or Python web-service. You can't do this with Airflow.
Code examples:
--------------
-Looking at the two examples below, you can see that Hamilton is a more lightweight and flexible way to define data pipelines.
-There is no scheduling information, etc required to run the code because Hamilton runs the pipeline in the same process as the
+Looking at the two examples below, you can see that Apache Hamilton is a more lightweight and flexible way to define data pipelines.
+There is no scheduling information, etc required to run the code because Apache Hamilton runs the pipeline in the same process as the
caller. This makes it easier to test and debug pipelines. Airflow, on the other hand, is a complete system for scheduling and
executing pipelines. It is more complex to set up and run. Note: If you stuck the contents of `run.py` in a function within
-the `example_dag.py`, the Hamilton pipeline could be used in the Airflow PythonOperator!
+the `example_dag.py`, the Apache Hamilton pipeline could be used in the Airflow PythonOperator!
-Hamilton:
+Apache Hamilton:
_________
-The below code here shows how you can define a simple data pipeline using Hamilton. The pipeline consists of three functions
+The below code here shows how you can define a simple data pipeline using Apache Hamilton. The pipeline consists of three functions
that are executed in sequence. The pipeline is defined in a module called `pipeline.py`, and then executed in a separate
script called `run.py`, which imports the pipeline module and executes it.
diff --git a/docs/code-comparisons/dagster.rst b/docs/code-comparisons/dagster.rst
index 83917f89..97c02c25 100644
--- a/docs/code-comparisons/dagster.rst
+++ b/docs/code-comparisons/dagster.rst
@@ -2,9 +2,9 @@
Dagster
=========
-Here are some code snippets to compare the macro orchestrator Dagster to the micro orchestrator Hamilton. Hamilton can run inside Dagster, but you wouldn't run Dagster inside Hamilton.
+Here are some code snippets to compare the macro orchestrator Dagster to the micro orchestrator Apache Hamilton. Apache Hamilton can run inside Dagster, but you wouldn't run Dagster inside Apache Hamilton.
-While the two have different scope, there's a lot of overlap between the two both in terms of functionality and API. Indeed, Dagster's software-defined assets introduced in 2022 matches Hamilton's declarative approach and should feel familiar to users of either.
+While the two have different scope, there's a lot of overlap between the two both in terms of functionality and API. Indeed, Dagster's software-defined assets introduced in 2022 matches Apache Hamilton's declarative approach and should feel familiar to users of either.
------
@@ -16,7 +16,7 @@ TL;DR
:header-rows: 1
* - Trait
- - Hamilton
+ - Apache Hamilton
- Dagster
* - Declarative API
- β
@@ -73,7 +73,7 @@ TL;DR
- `Lifecycle hooks <https://hamilton.dagworks.io/en/latest/reference/lifecycle-hooks/>`_ (easier to extend)
- `Op Hooks <https://docs.dagster.io/concepts/ops-jobs-graphs/op-hooks#op-hooks>`_
* - Plugins
- - `Spark <https://blog.dagworks.io/p/expressing-pyspark-transformations>`_, Dask, Ray, `Datadog <https://hamilton.dagworks.io/en/latest/reference/lifecycle-hooks/DDOGTracer/>`_, polars, pandera, and more (Hamilton is less restrictive and easier to extend)
+ - `Spark <https://blog.dagworks.io/p/expressing-pyspark-transformations>`_, Dask, Ray, `Datadog <https://hamilton.dagworks.io/en/latest/reference/lifecycle-hooks/DDOGTracer/>`_, polars, pandera, and more (Apache Hamilton is less restrictive and easier to extend)
- `Spark, Dask, polars, pandera, Databricks, Snowflake, Great Expections, and more <https://docs.dagster.io/integrations>`_ (Dagster integrations are more involved to develop)
* - Interactive Development
- `Jupyter Magic <https://hamilton.dagworks.io/en/latest/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic>`_, `VSCode extension <https://marketplace.visualstudio.com/items?itemName=ThierryJean.hamilton>`_
@@ -88,7 +88,7 @@ Dataflow definition
:align: left
+------------------------------------------------------------+----------------------------------------------------------+
- | Hamilton | Dagster |
+ | Apache Hamilton | Dagster |
+============================================================+==========================================================+
| .. literalinclude:: _dagster_snippets/hamilton_dataflow.py | .. literalinclude:: _dagster_snippets/dagster_dataflow.py|
| | |
@@ -102,7 +102,7 @@ Dataflow definition
:header-rows: 1
* - Trait
- - Hamilton
+ - Apache Hamilton
- Dagster
* - Define operations
- Uses the native Python function signature. The dataflow is assembled based on function/parameter names and type annotations.
@@ -125,7 +125,7 @@ Dataflow execution
:align: left
+-------------------------------------------------------------+------------------------------------------------------------+
- | Hamilton | Dagster |
+ | Apache Hamilton | Dagster |
+=============================================================+============================================================+
| .. literalinclude:: _dagster_snippets/hamilton_execution.py | .. literalinclude:: _dagster_snippets/dagster_execution.py |
| | |
@@ -136,7 +136,7 @@ Dataflow execution
:header-rows: 1
* - Trait
- - Hamilton
+ - Apache Hamilton
- Dagster
* - Execution instructions
- Define a ``Driver`` using the ``Builder`` object. It automatically assembles the graph from the dataflow definition found in ``dataflow.py``
@@ -156,7 +156,7 @@ Dataflow execution
More information
----------------
-For a full side-by-side example of Dagster and Hamilton, visit `this GitHub repository <https://github.com/apache/hamilton/tree/main/examples/dagster>`_
+For a full side-by-side example of Dagster and Apache Hamilton, visit `this GitHub repository <https://github.com/apache/hamilton/tree/main/examples/dagster>`_
For more questions, join our `Slack Channel <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_!
\ No newline at end of file
diff --git a/docs/code-comparisons/index.rst b/docs/code-comparisons/index.rst
index d7c6d9d9..b2b4deef 100644
--- a/docs/code-comparisons/index.rst
+++ b/docs/code-comparisons/index.rst
@@ -2,7 +2,7 @@
Code Comparisons
================
-This section showcases what Hamilton code looks like in comparison to other popular libraries and frameworks.
+This section showcases what Apache Hamilton code looks like in comparison to other popular libraries and frameworks.
.. toctree::
kedro
diff --git a/docs/code-comparisons/kedro.rst b/docs/code-comparisons/kedro.rst
index c4869b40..a12166ae 100644
--- a/docs/code-comparisons/kedro.rst
+++ b/docs/code-comparisons/kedro.rst
@@ -1,16 +1,16 @@
Kedro
=========
-Both ``Kedro`` and ``Hamilton`` are Python tools to help define directed acyclic graph (DAG) of data transformations. While there's overlap between the two in terms of features, we note two main differences:
+Both ``Kedro`` and ``Apache Hamilton`` are Python tools to help define directed acyclic graph (DAG) of data transformations. While there's overlap between the two in terms of features, we note two main differences:
-- ``Kedro`` is imperative and focuses on tasks; ``Hamilton`` is declarative and focuses on assets.
-- ``Kedro`` is heavier and comes with a project structure, YAML configs, and dataset definition to manage; ``Hamilton`` is lighter to adopt and you can progressively opt-in features that you find valuable.
+- ``Kedro`` is imperative and focuses on tasks; ``Apache Hamilton`` is declarative and focuses on assets.
+- ``Kedro`` is heavier and comes with a project structure, YAML configs, and dataset definition to manage; ``Apache Hamilton`` is lighter to adopt and you can progressively opt-in features that you find valuable.
On this page, we'll dive into these differences, compare features, and present
some code snippets from both tools.
.. note::
- See this `GitHub repository <https://github.com/apache/hamilton/tree/main/examples/kedro>`_ to compare a full project using Kedro or Hamilton.
+ See this `GitHub repository <https://github.com/apache/hamilton/tree/main/examples/kedro>`_ to compare a full project using Kedro or Apache Hamilton.
Imperative vs. Declarative
---------------------------
@@ -24,33 +24,33 @@ There are 3 steps to build and run a dataflow (a DAG, a data pipeline, etc.)
1. Define steps
~~~~~~~~~~~~~~~
-Imperative (``Kedro``) vs. declarative (``Hamilton``) leads to significant differences in **Step 2** and **Step 3** that will shape how you work with the tool. However, **Step 1** remains similar. In fact, both tools use the term **nodes** to refer to steps.
+Imperative (``Kedro``) vs. declarative (``Apache Hamilton``) leads to significant differences in **Step 2** and **Step 3** that will shape how you work with the tool. However, **Step 1** remains similar. In fact, both tools use the term **nodes** to refer to steps.
.. table::
:align: left
+---------------------------------------------------------+------------------------------------------------------------+
- | Kedro (imperative) | Hamilton (declarative) |
+ | Kedro (imperative) | Apache Hamilton (declarative) |
+=========================================================+============================================================+
| .. literalinclude:: _kedro_snippets/kedro_definition.py | .. literalinclude:: _kedro_snippets/hamilton_definition.py |
| | |
+---------------------------------------------------------+------------------------------------------------------------+
-The function implementations are exactly the same. Yet, notice that the function names and docstrings were edited slightly. Imperative approaches like ``Kedro`` typically refer to steps as *tasks* and prefer verbs to describe "the action of the function". Meanwhile, declarative approaches such as ``Hamilton`` describe steps as *assets* and use nouns to refer to "the value returned by the function". This might appear superficial, but it relates to the difference in **Step 2** and **Step 3**.
+The function implementations are exactly the same. Yet, notice that the function names and docstrings were edited slightly. Imperative approaches like ``Kedro`` typically refer to steps as *tasks* and prefer verbs to describe "the action of the function". Meanwhile, declarative approaches such as ``Apache Hamilton`` describe steps as *assets* and use nouns to refer to "the value returned by the function". This might appear superficial, but it relates to the difference in **Step 2** and [...]
2. Assemble dataflow
~~~~~~~~~~~~~~~~~~~~
With ``Kedro``, you need to take your functions from **Step 1** and create ``node`` objects, specifying the node's name, inputs, and outputs. Then, you create a ``pipeline`` from a set of ``nodes`` and ``Kedro`` assembles the nodes into a DAG. Imperative approaches need to specify how tasks (Kedro nodes) relate to each other.
-With ``Hamilton``, you pass the module containing all functions from **Step 1** and let Hamilton create the ``nodes`` and the ``dataflow``. This is possible because in declarative approaches like Hamilton, each function defines a transform **and** its dependencies on other functions. Notice how in **Step 1**, ``model_input_table()`` has parameters ``shuttles_preprocessed`` and ``companies_preprocessed``, which refers to other functions in the module. This contains all the required inform [...]
+With ``Apache Hamilton``, you pass the module containing all functions from **Step 1** and let Apache Hamilton create the ``nodes`` and the ``dataflow``. This is possible because in declarative approaches like Apache Hamilton, each function defines a transform **and** its dependencies on other functions. Notice how in **Step 1**, ``model_input_table()`` has parameters ``shuttles_preprocessed`` and ``companies_preprocessed``, which refers to other functions in the module. This contains al [...]
.. table::
:align: left
+---------------------------------------------------------+------------------------------------------------------------+
- | Kedro (imperative) | Hamilton (declarative) |
+ | Kedro (imperative) | Apache Hamilton (declarative) |
+=========================================================+============================================================+
| .. literalinclude:: _kedro_snippets/kedro_assemble.py | .. literalinclude:: _kedro_snippets/hamilton_assemble.py |
| | |
@@ -66,25 +66,25 @@ With ``Hamilton``, you pass the module containing all functions from **Step 1**
- Readability improves because you can understand how functions relate to each other without jumping between files.
-These benefits of ``Hamilton`` encourage developers to write smaller functions that are easier to debug and maintain, leading to major code quality gains. On the opposite, the burden of ``node`` and ``pipeline`` creation as projects grow in size lead to users stuffing more and more logic in a single node, making it increasingly harder to maintain.
+These benefits of ``Apache Hamilton`` encourage developers to write smaller functions that are easier to debug and maintain, leading to major code quality gains. On the opposite, the burden of ``node`` and ``pipeline`` creation as projects grow in size lead to users stuffing more and more logic in a single node, making it increasingly harder to maintain.
3. Execute dataflow
~~~~~~~~~~~~~~~~~~~~
The primary way to execute ``Kedro`` pipelines is to use the command line tool with ``kedro run --pipeline=my_pipeline``. Pipelines are typically designed for all nodes to be executed while reading data and writing results while going through nodes. It is closer to macro-orchestration frameworks like Airflow in spirit.
-On the opposite, ``Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Hamilton`` within a :doc:`FastAPI service <../integrations/fastapi>` service or to power an LLM application.
+On the opposite, ``Apache Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Apache Hamilton`` within a :doc:`FastAPI service <../integrations/fastapi>` service or to power an LLM application.
For comparable side-by-side code, we can dig into ``Kedro`` and use the ``SequentialRunner`` programmatically. To return pipeline results in-memory we would need to hack further with ``kedro.io.MemoryDataset``.
.. note::
- Hamilton also has rich support for I/O operations (see **Feature comparison** below)
+ Apache Hamilton also has rich support for I/O operations (see **Feature comparison** below)
.. table::
:align: left
+---------------------------------------------------------+------------------------------------------------------------+
- | Kedro (imperative) | Hamilton (declarative) |
+ | Kedro (imperative) | Apache Hamilton (declarative) |
+=========================================================+============================================================+
| .. literalinclude:: _kedro_snippets/kedro_execution.py | .. literalinclude:: _kedro_snippets/hamilton_execution.py |
| | |
@@ -92,9 +92,9 @@ For comparable side-by-side code, we can dig into ``Kedro`` and use the ``Sequen
An imperative pipeline like ``Kedro`` is a series of step, just like a recipe. The user can specify "from nodes" or "to nodes" to *slice* the pipeline and not have to execute it in full.
-For declarative dataflows like ``Hamilton`` you request assets / nodes by name and the tool will determine the required nodes to execute (here ``"model_input_table"``) avoiding wasteful compute.
+For declarative dataflows like ``Apache Hamilton`` you request assets / nodes by name and the tool will determine the required nodes to execute (here ``"model_input_table"``) avoiding wasteful compute.
-The simple Python interface provided by ``Hamilton`` allows you to potentially define and execute your dataflow from a single file, which is great to kickstart an analysis or project. Just use ``python dataflow.py`` to execute it!
+The simple Python interface provided by ``Apache Hamilton`` allows you to potentially define and execute your dataflow from a single file, which is great to kickstart an analysis or project. Just use ``python dataflow.py`` to execute it!
.. code-block:: python
@@ -145,7 +145,7 @@ The simple Python interface provided by ``Hamilton`` allows you to potentially d
Framework weight
----------------
-After imperative vs. declarative, the next largest difference is the type of user experience they provide. ``Kedro`` is a more opiniated and heavier framework; ``Hamilton`` is on the opposite end of the spectrum and tries to be the lightest library possible. This changes the learning curve, adoption, and how each tool will integrate with your stack.
+After imperative vs. declarative, the next largest difference is the type of user experience they provide. ``Kedro`` is a more opiniated and heavier framework; ``Apache Hamilton`` is on the opposite end of the spectrum and tries to be the lightest library possible. This changes the learning curve, adoption, and how each tool will integrate with your stack.
Kedro
~~~~~
@@ -159,17 +159,17 @@ Kedro
This provides guidance when building your first data pipeline, but it's also a lot to take in at once. As you'll see in the `project comparison on GitHub <https://github.com/apache/hamilton/tree/main/examples/kedro>`_, ``Kedro`` involves more files making it harder to navigate. Also, it's reliant on YAML which is `generally seen as an unreliable format <https://noyaml.com/>`_. If you have an existing data stack or favorite library, it might clash with ``Kedro``'s way of thing (e.g., you [...]
-Hamilton
+Apache Hamilton
~~~~~~~~
-``Hamilton`` attempts to get you started quickly. In fact, this page pretty much covered what you need to know:
+``Apache Hamilton`` attempts to get you started quickly. In fact, this page pretty much covered what you need to know:
- Define nodes and a dataflow using regular Python functions (no need to even import ``hamilton``!)
- Build a ``Driver`` with your dataflow module and call ``.execute()`` to get results
-``Hamilton`` allows you to start light and opt-in features as your project's requirements evolve (data validation, scaling compute, testing, etc.). Python is a powerful language with rich editor support and tooling hence why it advocates for "everything in Python" instead of external configs in YAML or JSON. For example, parameters, data assets, and configurations can very much live as dataclasses within a ``.py`` file. ``Hamilton`` was built with an extensive plugin system. There are ma [...]
+``Apache Hamilton`` allows you to start light and opt-in features as your project's requirements evolve (data validation, scaling compute, testing, etc.). Python is a powerful language with rich editor support and tooling hence why it advocates for "everything in Python" instead of external configs in YAML or JSON. For example, parameters, data assets, and configurations can very much live as dataclasses within a ``.py`` file. ``Apache Hamilton`` was built with an extensive plugin system [...]
-In fact, ``Hamilton`` is so lightweight, you could even run it inside ``Kedro``!
+In fact, ``Apache Hamilton`` is so lightweight, you could even run it inside ``Kedro``!
Feature comparison
------------------
@@ -180,7 +180,7 @@ Feature comparison
* - Trait
- Kedro
- - Hamilton
+ - Apache Hamilton
* - Focuses on
- Tasks (imperative)
- Assets (declarative)
@@ -207,7 +207,7 @@ Feature comparison
- Visualize entire dataflow, execution path, query what's upstream, etc. directly in a notebook or output to a file (``.png``, ``.svg``, etc.). Single dependency is ``graphviz``.
* - Interactive DAG viewer
- `Kedro Viz <https://github.com/kedro-org/kedro-viz>`_
- - `Hamilton UI <https://github.com/apache/hamilton/tree/main/ui>`_
+ - `Apache Hamilton UI <https://github.com/apache/hamilton/tree/main/ui>`_
* - Data validation
- `Community Pandera plugin <https://github.com/Galileo-Galilei/kedro-pandera/releases>`_
- `Native and Pandera plugin <https://hamilton.dagworks.io/en/latest/how-tos/run-data-quality-checks/>`_
@@ -231,7 +231,7 @@ Feature comparison
- β
-Both ``Kedro`` and ``Hamilton`` provide applications to view dataflows/pipelines and interact with their results. Here, ``Kedro`` provides a lighter webserver and UI, while ``Hamilton`` offers a production-ready containerized application.
+Both ``Kedro`` and ``Apache Hamilton`` provide applications to view dataflows/pipelines and interact with their results. Here, ``Kedro`` provides a lighter webserver and UI, while ``Apache Hamilton`` offers a production-ready containerized application.
.. list-table::
@@ -240,7 +240,7 @@ Both ``Kedro`` and ``Hamilton`` provide applications to view dataflows/pipelines
* - Trait
- Kedro Viz
- - Hamilton UI
+ - Apache Hamilton UI
* - Interactive dataflow viewer
- β
- β
@@ -276,6 +276,6 @@ Both ``Kedro`` and ``Hamilton`` provide applications to view dataflows/pipelines
More information
----------------
-For a full side-by-side example of Kedro and Hamilton, visit `this GitHub repository <https://github.com/apache/hamilton/tree/main/examples/kedro>`_
+For a full side-by-side example of Kedro and Apache Hamilton, visit `this GitHub repository <https://github.com/apache/hamilton/tree/main/examples/kedro>`_
For more questions, join our `Slack Channel <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_
diff --git a/docs/code-comparisons/langchain.rst b/docs/code-comparisons/langchain.rst
index d2dee605..ee27b518 100644
--- a/docs/code-comparisons/langchain.rst
+++ b/docs/code-comparisons/langchain.rst
@@ -3,15 +3,15 @@ LangChain
======================
Here we have some code snippets that help compare a vanilla code implementation
-with LangChain and Hamilton.
+with LangChain and Apache Hamilton.
LangChain's focus is on hiding details and making code terse.
-Hamilton's focus instead is on making code more readable, maintainable, and importantly customizeable.
+Apache Hamilton's focus instead is on making code more readable, maintainable, and importantly customizeable.
-So don't be surprised that Hamilton's code is "longer" - that's by design. There is
-also little abstraction between you, and the underlying libraries with Hamilton.
+So don't be surprised that Apache Hamilton's code is "longer" - that's by design. There is
+also little abstraction between you, and the underlying libraries with Apache Hamilton.
With LangChain they're abstracted away, so you can't really see easily what's going on
underneath.
@@ -25,7 +25,7 @@ A simple joke example
:align: left
+-----------------------------------------------------------+----------------------------------------------------------+-------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+===========================================================+==========================================================+=============================================================+
| .. literalinclude:: langchain_snippets/hamilton_invoke.py | .. literalinclude:: langchain_snippets/vanilla_invoke.py | .. literalinclude:: langchain_snippets/lcel_invoke.py |
| | | |
@@ -33,23 +33,23 @@ A simple joke example
.. figure:: langchain_snippets/hamilton-invoke.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 50%
- The Hamilton DAG visualized.
+ The Apache Hamilton DAG visualized.
-----------------------
A streamed joke example
-----------------------
-With Hamilton we can just swap the call function to return a streamed response.
+With Apache Hamilton we can just swap the call function to return a streamed response.
Note: you could use @config.when to include both streamed and non-streamed versions in the same DAG.
.. table:: Streamed Version
:align: left
+-------------------------------------------------------------+------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+=============================================================+============================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_streamed.py | .. literalinclude:: langchain_snippets/vanilla_streamed.py | .. literalinclude:: langchain_snippets/lcel_streamed.py |
| | | |
@@ -57,24 +57,24 @@ Note: you could use @config.when to include both streamed and non-streamed versi
.. figure:: langchain_snippets/hamilton-streamed.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 50%
- The Hamilton DAG visualized.
+ The Apache Hamilton DAG visualized.
-------------------------------
A "batch" parallel joke example
-------------------------------
In this batch example, the joke requests are parallelized.
-Note: with Hamilton you can delegate to many different backends for parallelization,
+Note: with Apache Hamilton you can delegate to many different backends for parallelization,
e.g. Ray, Dask, etc. We use multi-threading here.
.. table:: Batch Parallel Version
:align: left
+-------------------------------------------------------------+------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+=============================================================+============================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_batch.py | .. literalinclude:: langchain_snippets/vanilla_batch.py | .. literalinclude:: langchain_snippets/lcel_batch.py |
| | | |
@@ -82,24 +82,24 @@ e.g. Ray, Dask, etc. We use multi-threading here.
.. figure:: langchain_snippets/hamilton-batch.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 75%
- The Hamilton DAG visualized.
+ The Apache Hamilton DAG visualized.
----------------------
A "async" joke example
----------------------
-Here we show how to make the joke using async constructs. With Hamilton
+Here we show how to make the joke using async constructs. With Apache Hamilton
you can mix and match async and regular functions, the only change
-is that you need to use the async Hamilton Driver.
+is that you need to use the async Apache Hamilton Driver.
.. table:: Async Version
:align: left
+-------------------------------------------------------------+------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+=============================================================+============================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_async.py | .. literalinclude:: langchain_snippets/vanilla_async.py | .. literalinclude:: langchain_snippets/lcel_async.py |
| | | |
@@ -107,11 +107,11 @@ is that you need to use the async Hamilton Driver.
.. figure:: langchain_snippets/hamilton-async.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 50%
- The Hamilton DAG visualized.
+ The Apache Hamilton DAG visualized.
---------------------------------
@@ -125,7 +125,7 @@ that uses the different OpenAI model.
:align: left
+------------------------------------------------------------------+-----------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+==================================================================+=================================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_completion.py | .. literalinclude:: langchain_snippets/vanilla_completion.py | .. literalinclude:: langchain_snippets/lcel_completion.py |
| | | |
@@ -133,11 +133,11 @@ that uses the different OpenAI model.
.. figure:: langchain_snippets/hamilton-completion.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 50%
- The Hamilton DAG visualized with configuration provided for the completion path. Note the dangling node - that's normal, it's not used in the completion path.
+ The Apache Hamilton DAG visualized with configuration provided for the completion path. Note the dangling node - that's normal, it's not used in the completion path.
---------------------------------
@@ -152,7 +152,7 @@ to use Anthropic.
:align: left
+------------------------------------------------------------------+-----------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+==================================================================+=================================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_anthropic.py | .. literalinclude:: langchain_snippets/vanilla_anthropic.py | .. literalinclude:: langchain_snippets/lcel_anthropic.py |
| | | |
@@ -160,17 +160,17 @@ to use Anthropic.
.. figure:: langchain_snippets/hamilton-anthropic.png
- :alt: Structure of the Hamilton DAG
+ :alt: Structure of the Apache Hamilton DAG
:align: center
:width: 50%
- The Hamilton DAG visualized with configuration provided to use Anthropic.
+ The Apache Hamilton DAG visualized with configuration provided to use Anthropic.
---------------------------------
Logging
---------------------------------
-Here we show how to log more information about the joke request. Hamilton has
+Here we show how to log more information about the joke request. Apache Hamilton has
lots of customization options, and one out of the box is to log more information via
printing.
@@ -178,7 +178,7 @@ printing.
:align: left
+------------------------------------------------------------------+-----------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+==================================================================+=================================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_logging.py | .. literalinclude:: langchain_snippets/vanilla_logging.py | .. literalinclude:: langchain_snippets/lcel_logging.py |
| | | |
@@ -197,7 +197,7 @@ opinion it's better to be explicit about it.
:align: left
+------------------------------------------------------------------+-----------------------------------------------------------------+---------------------------------------------------------------+
- | Hamilton | Vanilla | LangChain |
+ | Apache Hamilton | Vanilla | LangChain |
+==================================================================+=================================================================+===============================================================+
| .. literalinclude:: langchain_snippets/hamilton_fallbacks.py | .. literalinclude:: langchain_snippets/vanilla_fallbacks.py | .. literalinclude:: langchain_snippets/lcel_fallbacks.py |
| | | |
diff --git a/docs/concepts/best-practices/code-organization.rst b/docs/concepts/best-practices/code-organization.rst
index 7ad4968b..fa2be2eb 100644
--- a/docs/concepts/best-practices/code-organization.rst
+++ b/docs/concepts/best-practices/code-organization.rst
@@ -2,9 +2,9 @@
Code Organization
=================
-Hamilton will force you to organize your code! Here's some tips.
+Apache Hamilton will force you to organize your code! Here's some tips.
-Hamilton forces you to put your code into modules that are distinct from where you run your code.
+Apache Hamilton forces you to put your code into modules that are distinct from where you run your code.
You'll soon find that a single python module does not make sense, and so you'll organically start to (very likely) put
like functions with like functions, i.e. thus creating domain specific modules --> `use this to your development
@@ -14,7 +14,7 @@ At Stitch Fix we:
#. Use modules to model team thinking, e.g. date\_features.py.
#. Use modules to helps isolate what youβre working on.
-#. Use modules to replace parts of your Hamilton dataflow very easily for different contexts.
+#. Use modules to replace parts of your Apache Hamilton dataflow very easily for different contexts.
Team thinking
-------------
@@ -42,7 +42,7 @@ want to swap/change/augment an input, having a function that would map to it def
lot of flexibility. Rather than having a single module with all functions defined in it, separating the functions into
different modules could be a productivity win.
-Why? That's because when you come to tell Hamilton what functions constitute your dataflow (i.e. DAG), you'll be able
+Why? That's because when you come to tell Apache Hamilton what functions constitute your dataflow (i.e. DAG), you'll be able
to simply replace/add/change the module being passed. So if you want to compute inputs for certain functions
differently, this composability of including/excluding modules, when building the DAG provides a lot of flexibility
that you can exploit to make your development cycle faster.
diff --git a/docs/concepts/best-practices/common-indices.rst b/docs/concepts/best-practices/common-indices.rst
index fa052ab4..a5237fbd 100644
--- a/docs/concepts/best-practices/common-indices.rst
+++ b/docs/concepts/best-practices/common-indices.rst
@@ -4,14 +4,14 @@ Common Indices
If you're creating dataframes, then this will apply to you!
-While Hamilton is a general-purpose framework, we've found a common pattern is to manipulate datasets that have shared
+While Apache Hamilton is a general-purpose framework, we've found a common pattern is to manipulate datasets that have shared
indices (spines) for creating dataframes.
Although this might not apply towards every use-case (E.G. more complex joins with spark dataframes), a large selection
of use-cases can be enabled if every dataframe in your pipeline shares an index. This is particularly pertinent when
writing transformations over (non-event-based) time-series data.
-While Hamilton currently has no means of enforcing shared-spine, it is up to the writer of the function to validate
+While Apache Hamilton currently has no means of enforcing shared-spine, it is up to the writer of the function to validate
input data as necessary. Thus we recommend the following if you are creating a dataframe as output:
Best practice:
diff --git a/docs/concepts/best-practices/function-naming.rst b/docs/concepts/best-practices/function-naming.rst
index b1aeb329..cda48577 100644
--- a/docs/concepts/best-practices/function-naming.rst
+++ b/docs/concepts/best-practices/function-naming.rst
@@ -4,14 +4,14 @@ Function Naming
Here are three important points about function naming:
-#. It enables you to define your Hamilton dataflow.
+#. It enables you to define your Apache Hamilton dataflow.
#. It drives collaboration & code reuse.
#. It serves as documentation itself.
-You don't need to get this right the first time -- search and replace is really easy with Hamilton code bases -- but it
+You don't need to get this right the first time -- search and replace is really easy with Apache Hamilton code bases -- but it
is something to converge thinking on!
-It enables you to define your Hamilton dataflow
+It enables you to define your Apache Hamilton dataflow
-----------------------------------------------
As the name of a hamilton function defines the name of the created artifact, naming is vital to a readable, extensible
@@ -24,7 +24,7 @@ hamilton codebase. Names must mean something:
...
In this case, ``foo_bar`` is not helpful - it's unclear what this function produces at all. Remember you want function
-names to mean something, since that will enable clarity when using Hamilton, what is being requested, and will help
+names to mean something, since that will enable clarity when using Apache Hamilton, what is being requested, and will help
document what the function itself is doing.
It drives collaboration and reuse
@@ -35,7 +35,7 @@ When people come to encounter your code, they'll need to understand it, add to i
You'll want to ensure some standardization to enable:
#. Mapping business concepts to function names. E.g. That will help people to find things in the code that map to things that happen within your business.
-#. Ensuring naming uniformity across the code base. People usually follow the precedent of the code around them, so if everything in a particular module for say, date features, has a ``D_`` prefix, then they will likely follow that naming convention. This is likely something you will iterate on -- and it's best to try to converge on a team naming convention once you have a feel for the Hamilton functions being written by the team.
+#. Ensuring naming uniformity across the code base. People usually follow the precedent of the code around them, so if everything in a particular module for say, date features, has a ``D_`` prefix, then they will likely follow that naming convention. This is likely something you will iterate on -- and it's best to try to converge on a team naming convention once you have a feel for the Apache Hamilton functions being written by the team.
We suggest that long functions names that are separated by ``_`` aren't a bad thing. E.g. if you were to come across a
function named ``life_time_value`` versus ``ltv`` versus ``l_t_v``, which one is more obvious as to what it is and what
diff --git a/docs/concepts/best-practices/index.rst b/docs/concepts/best-practices/index.rst
index d7dd760d..13eb0e48 100644
--- a/docs/concepts/best-practices/index.rst
+++ b/docs/concepts/best-practices/index.rst
@@ -1,7 +1,7 @@
==============
Best Practices
==============
-A set of best-practices to help you get the most out of Hamilton quickly and easily.
+A set of best-practices to help you get the most out of Apache Hamilton quickly and easily.
.. toctree::
diff --git a/docs/concepts/best-practices/loading-data.rst b/docs/concepts/best-practices/loading-data.rst
index 601348ce..57557f91 100644
--- a/docs/concepts/best-practices/loading-data.rst
+++ b/docs/concepts/best-practices/loading-data.rst
@@ -2,7 +2,7 @@
Loading Data
============
-In Hamilton, data loaders are just the same as other functions in the DAG. They take in configuration parameters, and
+In Apache Hamilton, data loaders are just the same as other functions in the DAG. They take in configuration parameters, and
output datasets in the desired form. Following up on the marketing spend dataset, you might write a data loader that
reads a dataframe saved in csv format on s3 like this:
@@ -32,11 +32,11 @@ go. However, there are a few considerations you might have prior to productiona
Plugging in new Data Sources
----------------------------
-An advantage of Hamilton is that it allows for rapid plug-and play for various components of your pipeline. This is
+An advantage of Apache Hamilton is that it allows for rapid plug-and play for various components of your pipeline. This is
particularly important for data loading, where you might want to load your data from different sources depending on some
context. For instance -- if you're running your pipeline in production, you may want to use the production data sources.
If you're running it in QA, you might want to use the staging data sources. Or, if you're running it locally, you might
-want to use abbreviated, in-memory data sources for testing. While Hamilton is not opinionated on exactly _how_ you make
+want to use abbreviated, in-memory data sources for testing. While Apache Hamilton is not opinionated on exactly _how_ you make
this switch, it presents a variety of tooling that can make it more manageable. Some options. To demonstrate some
techniques, let's continue on the example of loading marketing spend...
@@ -104,5 +104,5 @@ Then you can invoke your driver but set the config differently:
data_loaders, ...)
Note that there are a variety of other ways you can organize your code -- at this point its entirely use-case dependent.
-Hamilton is a language for declaring dataflows that's applicable towards a multitude of use-cases. It's not going to
+Apache Hamilton is a language for declaring dataflows that's applicable towards a multitude of use-cases. It's not going to
dictate how to write your functions or where you put them.
diff --git a/docs/concepts/best-practices/migrating-to-hamilton.rst b/docs/concepts/best-practices/migrating-to-hamilton.rst
index 738ff3ef..e1384a73 100644
--- a/docs/concepts/best-practices/migrating-to-hamilton.rst
+++ b/docs/concepts/best-practices/migrating-to-hamilton.rst
@@ -1,8 +1,8 @@
=====================
-Migrating to Hamilton
+Migrating to Apache Hamilton
=====================
-Here are two suggestions for helping you migrate to Hamilton
+Here are two suggestions for helping you migrate to Apache Hamilton
Continuous Integration for Comparisons
--------------------------------------
@@ -11,21 +11,21 @@ Create a way to easily & frequently compare results.
#. Integrate with continuous integration (CI) system if you can.
#. ππ Having a means that tests code early & often will helps diagnose bugs in your old code (most likely) or your new implementation (less likely).
-#. Specifically, have a system to compare the output of your Hamilton code, to compare to the output of your existing system.
+#. Specifically, have a system to compare the output of your Apache Hamilton code, to compare to the output of your existing system.
.. image:: ../../_static/Hamilton_ApplyMeetup_2022_migration_CI_1.svg
- :alt: Example CI process that we used at Stitch Fix for migrating to Hamilton
+ :alt: Example CI process that we used at Stitch Fix for migrating to Apache Hamilton
Integrate into your code base via a `"custom wrapper object"`
-------------------------------------------------------------
-If you have existing systems that you want to integrate Hamilton into, it might require non-trivial effort for you to
-change those systems to be able to use Hamilton. If that's the case, then we suggest creating a "custom object" to
-"wrap" Hamilton, so that it's easier to migrate to it.
+If you have existing systems that you want to integrate Apache Hamilton into, it might require non-trivial effort for you to
+change those systems to be able to use Apache Hamilton. If that's the case, then we suggest creating a "custom object" to
+"wrap" Apache Hamilton, so that it's easier to migrate to it.
Specifically, this custom wrapper object class's purpose is to match your existing API expectations. It will act as the
-translation layer from your existing API expectations, to what running Hamilton requires, and back. In Hamilton
-terminology, this is a `Custom Driver Wrapper`, since it wraps around the Hamilton Driver class.
+translation layer from your existing API expectations, to what running Apache Hamilton requires, and back. In Apache Hamilton
+terminology, this is a `Custom Driver Wrapper`, since it wraps around the Apache Hamilton Driver class.
.. image:: ../../_static/Hamilton_ApplyMeetup_2022_wrapper.svg
:alt: The wrapper driver class helps ensure your existing API expectations are matched.
@@ -33,4 +33,4 @@ terminology, this is a `Custom Driver Wrapper`, since it wraps around the Hamilt
This is a best practice because:
#. When migrating, it's best to avoid making too many changes. So don't change your API expectations if you can.
-#. It allows you to easily insert Hamilton into any context. Thereby minimizing potential migration problems.
+#. It allows you to easily insert Apache Hamilton into any context. Thereby minimizing potential migration problems.
diff --git a/docs/concepts/best-practices/output-immutability.rst b/docs/concepts/best-practices/output-immutability.rst
index 2d44b8b9..4bdf218c 100644
--- a/docs/concepts/best-practices/output-immutability.rst
+++ b/docs/concepts/best-practices/output-immutability.rst
@@ -2,17 +2,17 @@
Output Immutability
===================
-In Hamilton, functions are only called once!
+In Apache Hamilton, functions are only called once!
Immutability means, that once a "data structure", e.g. a column is created, and output by a function, the values in the
column are not changeable.
-When Hamilton figures out the execution call path, it walks it and calls functions only once. This means, that if the
+When Apache Hamilton figures out the execution call path, it walks it and calls functions only once. This means, that if the
output of a function is immutable, then there's only one place it was created; it's not modified anywhere else. This
provides a great debugging experience if there are ever issues in your dataflow. We believe that by default, one should
always strive for immutability of outputs.
-However, it is up to you, the Hamilton function writer, to ensure that immutability is something that is adhered to.
+However, it is up to you, the Apache Hamilton function writer, to ensure that immutability is something that is adhered to.
Best practice:
--------------
diff --git a/docs/concepts/best-practices/using-within-your-etl-system.rst b/docs/concepts/best-practices/using-within-your-etl-system.rst
index 7f169104..444b9a48 100644
--- a/docs/concepts/best-practices/using-within-your-etl-system.rst
+++ b/docs/concepts/best-practices/using-within-your-etl-system.rst
@@ -2,7 +2,7 @@
Using within your ETL System
============================
-Conceptually you can integrate Hamilton within your existing ETL system quite easily:
+Conceptually you can integrate Apache Hamilton within your existing ETL system quite easily:
Compatibility Matrix
--------------------
@@ -34,8 +34,8 @@ Compatibility Matrix
ETL Recipe
----------
-#. Write Hamilton functions & `βdriverβ` code.
-#. Publish your Hamilton functions in a package, or import via other means (e.g. checkout a repository & include in python path).
+#. Write Apache Hamilton functions & `βdriverβ` code.
+#. Publish your Apache Hamilton functions in a package, or import via other means (e.g. checkout a repository & include in python path).
#. Include `sf-hamilton` as a python dependency
#. Have your ETL system execute your βdriverβ code.
#. Profit.
diff --git a/docs/concepts/builder.rst b/docs/concepts/builder.rst
index a20becdd..e5955a7d 100644
--- a/docs/concepts/builder.rst
+++ b/docs/concepts/builder.rst
@@ -75,7 +75,7 @@ It encourages organizing code into logical modules (e.g., feature processing, mo
.. note::
- Your modules may have same named functions which will raise an error when using ``.build()`` since we cannot have two nodes with the same name. You can use the method ``.allow_module_overrides()`` and Hamilton will choose the function from the later imported module.
+ Your modules may have same named functions which will raise an error when using ``.build()`` since we cannot have two nodes with the same name. You can use the method ``.allow_module_overrides()`` and Apache Hamilton will choose the function from the later imported module.
@@ -88,7 +88,7 @@ It encourages organizing code into logical modules (e.g., feature processing, mo
.build()
)
- If ``module_A`` and ``module_B`` both have the function ``foo()``, Hamilton will use ``module_B.foo()`` when constructing the DAG. See https://github.com/apache/hamilton/tree/main/examples/module_overrides for more info.
+ If ``module_A`` and ``module_B`` both have the function ``foo()``, Apache Hamilton will use ``module_B.foo()`` when constructing the DAG. See https://github.com/apache/hamilton/tree/main/examples/module_overrides for more info.
with_config()
-------------
@@ -182,7 +182,7 @@ Adds `DataSaver` and `DataLoader` nodes to your dataflow. This allows to visuali
with_cache()
--------------
-This enables Hamilton's caching feature, which allows to automatically store intermediary results and reuse them in subsequent executions to skip computations. Learn more in the :doc:`/concepts/caching` section.
+This enables Apache Hamilton's caching feature, which allows to automatically store intermediary results and reuse them in subsequent executions to skip computations. Learn more in the :doc:`/concepts/caching` section.
.. code-block:: python
@@ -216,7 +216,7 @@ This allows to add multiple Lifecycle hooks to the Driver. This is a very flexi
.build()
)
-Other hooks are available to output a progress bar in the terminal, do experiment tracking for your Hamilton runs, cache results to disk, send logs to DataDog, and more!
+Other hooks are available to output a progress bar in the terminal, do experiment tracking for your Apache Hamilton runs, cache results to disk, send logs to DataDog, and more!
enable_dynamic_execution()
--------------------------
diff --git a/docs/concepts/caching.rst b/docs/concepts/caching.rst
index 29b2c965..33aa6b64 100644
--- a/docs/concepts/caching.rst
+++ b/docs/concepts/caching.rst
@@ -76,12 +76,12 @@ By traversing the cache keys' ``dependencies_data_versions``, we can actually re
.. warning::
- Cache keys could be unstable across Python and Hamilton versions (because of new features, bug fixes, etc.). Upgrading Python or Hamilton could require starting with a new empty cache for reliable behavior.
+ Cache keys could be unstable across Python and Apache Hamilton versions (because of new features, bug fixes, etc.). Upgrading Python or Apache Hamilton could require starting with a new empty cache for reliable behavior.
Observing the cache
-------------------
-Caching is best understood throung interacting with it. Hamilton offers many utilities to observe and introspect the cache manually.
+Caching is best understood throung interacting with it. Apache Hamilton offers many utilities to observe and introspect the cache manually.
Logging
~~~~~~~
@@ -321,7 +321,7 @@ Code version
The ``code_version`` of a node is determined by hashing its source code, ignoring docstring and comments.
-Importantly, Hamilton will not version nested function calls. If you edit utility functions or upgrade Python libraries, the cache might incorrectly assume the code to be the same.
+Importantly, Apache Hamilton will not version nested function calls. If you edit utility functions or upgrade Python libraries, the cache might incorrectly assume the code to be the same.
For example, take the following function ``foo``:
@@ -417,7 +417,7 @@ Storage
The caching feature is powered by two data storages:
-- **Metadata store**: It contains information about past ``Driver`` executions (**code version**, **data version**, run id, etc.). From this metadata, Hamilton determines if a node needs to be executed or not. This metadata is generally lightweight.
+- **Metadata store**: It contains information about past ``Driver`` executions (**code version**, **data version**, run id, etc.). From this metadata, Apache Hamilton determines if a node needs to be executed or not. This metadata is generally lightweight.
- **Result store**: It's a key-value store that maps a **data version** to a **result**. It's completely unaware of nodes, executions, etc. and simply holds the **results**. The result store can significantly grow in size depending on your usage. By default, all results are pickled, but :ref:`other formats are possible <cache-result-format>`.
@@ -453,7 +453,7 @@ Centralizing your cache by project is useful when you have nodes that are reused
Globally
^^^^^^^^^^
-Using a global cache is easier storage management. Since the metadata and the results for *all* your Hamilton dataflows are in one place, it can be easier to cleanup disk space.
+Using a global cache is easier storage management. Since the metadata and the results for *all* your Apache Hamilton dataflows are in one place, it can be easier to cleanup disk space.
.. code-block:: python
@@ -657,12 +657,12 @@ This snippet loads in-memory data from persisted metadata and result stores. The
Roadmap
-------
-Caching is a significant Hamilton feature and there are plans to expand it. Here are some ideas and areas for development. Feel free comment on them or make other suggestions via `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ or GitHub!
+Caching is a significant Apache Hamilton feature and there are plans to expand it. Here are some ideas and areas for development. Feel free comment on them or make other suggestions via `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ or GitHub!
-- **Hamilton UI integration**: caching introduces the concept of ``data_version``. This metadata could be captured by the Hamilton UI to show how different values are used across dataflow executions. This would be particularly useful for experiment tracking and lineage.
+- **Apache Hamilton UI integration**: caching introduces the concept of ``data_version``. This metadata could be captured by the Apache Hamilton UI to show how different values are used across dataflow executions. This would be particularly useful for experiment tracking and lineage.
- **Distributed caching support**: the initial release supports multithreading and multiprocessing on a single machine. For distributed execution, we will need ``ResultStore`` and ``MetadataStore`` that can be remote and are safe for concurrent access.
- **Integrate with remote execution** (Ray, Skypilot, Modal, Runhouse): facilitate a pattern where the dataflow is executed locally, but some nodes can selectively be executed remotely and have their results cached locally.
- **async support**: Support caching with ``AsyncDriver``. This requires a significant amount of code, but the core logic shouldn't change much.
- **cache eviction**: Allow to set up a max storage (in size or number of items) or time-based policy to delete data from the metadata and result stores. This would help with managing the cache size.
- **more store backends**: The initial release includes backend supported by the Python standard library (SQLite metadata and file-based results). Could support more backends via `fsspec <https://filesystem-spec.readthedocs.io/en/latest/?badge=latest>`_ (AWS, Azure, GCP, Databricks, etc.)
-- **support more types**: Include specialized hashing functions for complex objects from popular libraries. This can be done through Hamilton extensions.
+- **support more types**: Include specialized hashing functions for complex objects from popular libraries. This can be done through Apache Hamilton extensions.
diff --git a/docs/concepts/driver.rst b/docs/concepts/driver.rst
index 85a58ed0..1ddc4fc5 100644
--- a/docs/concepts/driver.rst
+++ b/docs/concepts/driver.rst
@@ -2,7 +2,7 @@
Driver
======
-Once you defined your dataflow in a Python module, you need to create a Hamilton Driver to execute it. This page details the Driver basics, which include:
+Once you defined your dataflow in a Python module, you need to create a Apache Hamilton Driver to execute it. This page details the Driver basics, which include:
1. Defining the Driver
2. Visualizing the dataflow
@@ -44,7 +44,7 @@ The most basic Driver is built like this:
The ``.build()`` method will fail if the definition found in ``my_dataflow`` is invalid (e.g., type mismatch, missing annotations) allowing you to fix issues and iterate quickly.
-The ``Driver`` is defined in the context you intend to run, separately from your dataflow module. It can be in a script, notebook, server, web app, or anywhere else Python can run. As a convention, most Hamilton code examples use a script named ``run.py``.
+The ``Driver`` is defined in the context you intend to run, separately from your dataflow module. It can be in a script, notebook, server, web app, or anywhere else Python can run. As a convention, most Apache Hamilton code examples use a script named ``run.py``.
Visualize the dataflow
----------------------
@@ -93,7 +93,7 @@ The Driver automatically determines the minimum required path to compute request
Development tips
----------------
-With Hamilton, development time is mostly spent writing functions for your dataflow in a Python module. Rebuilding the Driver and visualizing your dataflow as you make changes helps iterative development. Find below two useful development workflows.
+With Apache Hamilton, development time is mostly spent writing functions for your dataflow in a Python module. Rebuilding the Driver and visualizing your dataflow as you make changes helps iterative development. Find below two useful development workflows.
With a Python module
~~~~~~~~~~~~~~~~~~~~
@@ -160,7 +160,7 @@ Recap
Next step
---------
-Now, you know the basics of authoring and executing Hamilton dataflows! We encourage you to:
+Now, you know the basics of authoring and executing Apache Hamilton dataflows! We encourage you to:
- Write some code with our `interactive tutorials <https://www.tryhamilton.dev/intro>`_
- Kickstart your project with `community dataflows <https://hub.dagworks.io/docs/>`_
diff --git a/docs/concepts/function-modifiers-advanced.rst b/docs/concepts/function-modifiers-advanced.rst
index 35ba02df..e8b051b7 100644
--- a/docs/concepts/function-modifiers-advanced.rst
+++ b/docs/concepts/function-modifiers-advanced.rst
@@ -10,7 +10,7 @@ The page :doc:`function-modifiers` details how to use decorators to write expres
Nonetheless, there exists higher level abstractions for power users that *may* be useful for integrations with your existing platform. If you want to use complex machinery instead of writing 1 additional function, comeback when it's your 10th manually addition π
-This page assumes an advanced understanding of Hamilton and will cover:
+This page assumes an advanced understanding of Apache Hamilton and will cover:
- @pipe
- @subdag
diff --git a/docs/concepts/function-modifiers.rst b/docs/concepts/function-modifiers.rst
index c3b38c50..0e16c781 100644
--- a/docs/concepts/function-modifiers.rst
+++ b/docs/concepts/function-modifiers.rst
@@ -2,9 +2,9 @@
Function modifiers
==================
-In :doc:`node`, we discussed how to write Python functions to define Hamilton nodes and dataflow. In the basic case, each function defines one node.
+In :doc:`node`, we discussed how to write Python functions to define Apache Hamilton nodes and dataflow. In the basic case, each function defines one node.
-Yet, it's common to need nodes with similar purposes but different dependencies, such as preprocessing training and evaluation datasets. In that case, using a **function modifier** can help create both nodes from a single Hamilton function!
+Yet, it's common to need nodes with similar purposes but different dependencies, such as preprocessing training and evaluation datasets. In that case, using a **function modifier** can help create both nodes from a single Apache Hamilton function!
On this page, you'll learn:
@@ -20,9 +20,9 @@ This page covers important conceptual notions but is not exhaustive. To find det
Decorators
----------
-Python decorators are statements that begin with ``@`` located above function definitions. Hamilton uses decorators to implement function modifiers and reduce the amount of code you have to write to make expressive dataflows.
+Python decorators are statements that begin with ``@`` located above function definitions. Apache Hamilton uses decorators to implement function modifiers and reduce the amount of code you have to write to make expressive dataflows.
-Multiple decorators can be stacked on a single function and are applied from bottom to top. Hamilton decorators should be insensitive to ordering, but be careful with non-Hamilton decorators (e.g., @retries, @time). See this `decorator primer <https://realpython.com/primer-on-python-decorators/>`_ to learn more.
+Multiple decorators can be stacked on a single function and are applied from bottom to top. Apache Hamilton decorators should be insensitive to ordering, but be careful with non-Apache Hamilton decorators (e.g., @retries, @time). See this `decorator primer <https://realpython.com/primer-on-python-decorators/>`_ to learn more.
Function modifiers were designed to have clear semantics, so you should be able to figure out what they do from their name. For instance, the following code adds metadata using ``@tag`` and conducts some checks over the return value with ``check_output``.
@@ -41,7 +41,7 @@ Function modifiers were designed to have clear semantics, so you should be able
Reminder: Anatomy of a node
---------------------------
-This section from the page :doc:`node` details how a Python function maps to a Hamilton node. We'll reuse these terms to explain the function modifiers.
+This section from the page :doc:`node` details how a Python function maps to a Apache Hamilton node. We'll reuse these terms to explain the function modifiers.
.. image:: ../_static/function_anatomy.png
:scale: 13%
@@ -75,7 +75,7 @@ Add metadata to a node
@tag
~~~~~~~~
-The ``@tag`` decorator **doesn't modify the function/node**. It attaches metadata to the node that can be used by Hamilton and you. It can help tag nodes by ownership, data source, version, infrastructure, and anything else.
+The ``@tag`` decorator **doesn't modify the function/node**. It attaches metadata to the node that can be used by Apache Hamilton and you. It can help tag nodes by ownership, data source, version, infrastructure, and anything else.
For example, this tags the associated data product and the sensitivity of the data.
@@ -161,7 +161,7 @@ The next snippet checks if the returned Series is of type ``np.int32``, which is
- To see all available validators, go to the file ``hamilton/data_quality/default_validators.py`` and view the variable ``AVAILABLE_DEFAULT_VALIDATORS``.
-- The function modifier ``@check_output_custom`` allows you to define your own validator. Validators inherit the ``base.BaseDefaultValidator`` class and are essentially standardized Hamilton node definitions (instead of functions). See ``hamilton/data_quality/default_validators.py`` or reach out on `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ for help!
+- The function modifier ``@check_output_custom`` allows you to define your own validator. Validators inherit the ``base.BaseDefaultValidator`` class and are essentially standardized Apache Hamilton node definitions (instead of functions). See ``hamilton/data_quality/default_validators.py`` or reach out on `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ for help!
- Note: ``@check_output_custom`` decorators cannot be stacked, but they instead can take multiple validators.
.. note::
@@ -173,13 +173,13 @@ The next snippet checks if the returned Series is of type ``np.int32``, which is
pandera support
~~~~~~~~~~~~~~~
-Hamilton has a pandera plugin for data validation that you can install with ``pip install sf-hamilton[pandera]``. Then, you can pass a pandera schema (for DataFrame or Series) to ``@check_output(schema=...)``.
+Apache Hamilton has a pandera plugin for data validation that you can install with ``pip install sf-hamilton[pandera]``. Then, you can pass a pandera schema (for DataFrame or Series) to ``@check_output(schema=...)``.
pydantic support
~~~~~~~~~~~~~~~~
-Hamilton also supports data validation of pydantic models, which can be enabled with ``pip install sf-hamilton[pydantic]``. With pydantic installed, you can pass any subclass of the pydantic base model to ``@check_output(model=...)``. Pydantic validation is performed in strict mode, meaning that raw values will not be coerced to the model's types. For more information on strict mode see the `pydantic docs <https://docs.pydantic.dev/latest/concepts/strict_mode/>`_.
+Apache Hamilton also supports data validation of pydantic models, which can be enabled with ``pip install sf-hamilton[pydantic]``. With pydantic installed, you can pass any subclass of the pydantic base model to ``@check_output(model=...)``. Pydantic validation is performed in strict mode, meaning that raw values will not be coerced to the model's types. For more information on strict mode see the `pydantic docs <https://docs.pydantic.dev/latest/concepts/strict_mode/>`_.
Split node output into *n* nodes
@@ -296,7 +296,7 @@ For the decorator, you must specify one or more ``key=value`` pairs. Then, you n
This example uses ``@config.when()`` to select between a binary classifier and a regressor model. Notice a few elements:
-- both functions have the same name ``base_model`` with a suffix ``__binary`` or ``__regression``. This is required because Python enforces unique function names. After the config determines which function to load, Hamilton will remove the suffix from the node name.
+- both functions have the same name ``base_model`` with a suffix ``__binary`` or ``__regression``. This is required because Python enforces unique function names. After the config determines which function to load, Apache Hamilton will remove the suffix from the node name.
- the two functions have different return types, so ``train_model`` needs to annotate ``base_model`` as a ``Union[]`` type.
.. code-block:: python
@@ -370,9 +370,9 @@ Load and save external data
Most dataflows require reading or writing data to external sources in some capacity. It's a good idea to conduct this step in a node separated from transformations to trace failures more easily.
-Nevertheless, adding one function per read/write becomes tedious and hard to maintain. Hamilton provides well-tested implementations for common formats (JSON, CSV, Parquet, etc.) available through ``@load_from`` and ``@save_to`` decorators and materializers (see :doc:`materialization`).
+Nevertheless, adding one function per read/write becomes tedious and hard to maintain. Apache Hamilton provides well-tested implementations for common formats (JSON, CSV, Parquet, etc.) available through ``@load_from`` and ``@save_to`` decorators and materializers (see :doc:`materialization`).
-More formats are available through Hamilton plugins, and you should be able to add your own custom loader/saver (reach out on `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ for help!)
+More formats are available through Apache Hamilton plugins, and you should be able to add your own custom loader/saver (reach out on `Slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_ for help!)
@load_from
~~~~~~~~~~
diff --git a/docs/concepts/glossary.rst b/docs/concepts/glossary.rst
index 3addd578..b04f578b 100644
--- a/docs/concepts/glossary.rst
+++ b/docs/concepts/glossary.rst
@@ -16,13 +16,13 @@ Before we dive into the concepts, let's clarify the terminology we'll be using:
the other), acyclic, (there are no cycles, i.e., no function runs before itself), and a graph (it is easily \
naturally represented by nodes and edges) and can be represented visually. See :doc:`node`.
* - Node |
- Hamilton node |
+ Apache Hamilton node |
Transform
- A single step in the dataflow DAG representing a computation -- usually 1:1 with functions but decorators break that \
pattern -- in which case multiple transforms trace back to a single function. See :doc:`node`.
* - Function |
Python function |
- Hamilton function |
+ Apache Hamilton function |
Node definition
- A Python function written by a user to create a single node (in the standard case) or \
many (using function modifiers). See :doc:`node`.
@@ -30,7 +30,7 @@ Before we dive into the concepts, let's clarify the terminology we'll be using:
Python module
- Python code organized into a ``.py`` file. These are natural groupings of functions that turn to a set of nodes. See :doc:`best-practices/code-organization` for more details.
* - Driver |
- Hamilton Driver
+ Apache Hamilton Driver
- An object that loads Python modules to build a dataflow. It is responsible for visualizing and executing the \
dataflow. See :doc:`driver`.
* - script |
@@ -41,4 +41,4 @@ Before we dive into the concepts, let's clarify the terminology we'll be using:
- Data that dictates the way the DAG is constructed. See :doc:`driver`.
* - Function modifiers |
Decorators
- - A function that modifies how your Hamilton function is compiled into a Hamilton node. See :doc:`function-modifiers`.
+ - A function that modifies how your Apache Hamilton function is compiled into a Apache Hamilton node. See :doc:`function-modifiers`.
diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst
index dceaa082..c27e1ec4 100644
--- a/docs/concepts/index.rst
+++ b/docs/concepts/index.rst
@@ -3,7 +3,7 @@ Concepts
====================
Now that you're familiar with the basics and have run your own dataflow, let's dive into the
-concepts that makes Hamilton unique and powerful.
+concepts that makes Apache Hamilton unique and powerful.
.. toctree::
diff --git a/docs/concepts/materialization.rst b/docs/concepts/materialization.rst
index e8f7b9b9..b12d7680 100644
--- a/docs/concepts/materialization.rst
+++ b/docs/concepts/materialization.rst
@@ -6,7 +6,7 @@ So far, we executed our dataflow using the ``Driver.execute()`` method, which ca
On this page, you'll learn:
-- How to load and save data in Hamilton
+- How to load and save data in Apache Hamilton
- Why use materialization
- What are ``DataSaver`` and ``DataLoader`` objects
- The difference between ``.execute()`` and ``.materialize()``
@@ -42,14 +42,14 @@ Without materialization
Observations:
-1. These two approaches load and save data using ``pandas`` and ``xgboost`` without any Hamilton constructs. These methods are transparent and simple to get started, but as the number of node grows (or across projects) defining one node per parquet file to load introduces a lot of boilerplate.
+1. These two approaches load and save data using ``pandas`` and ``xgboost`` without any Apache Hamilton constructs. These methods are transparent and simple to get started, but as the number of node grows (or across projects) defining one node per parquet file to load introduces a lot of boilerplate.
2. Using **1) from nodes** improves visibility by including loading & saving in the dataflow (as illustrated).
3. Using **2) from ``Driver``** facilitates modifying loading & saving before code execution when executing the code, without modifying the dataflow itself. It is particularly useful when moving from development to production.
Limitations
~~~~~~~~~~~~
-Hamilton's approach to "materializations" aims to solve 3 limitations:
+Apache Hamilton's approach to "materializations" aims to solve 3 limitations:
1. **Redundancy**: deduplicate loading & saving code to improve maintainability and debugging
2. **Observability**: include loading & saving in the dataflow for full observability and allow hooks
@@ -80,15 +80,15 @@ the :doc:`@datasaver() <../reference/decorators/datasaver/>` and :doc:`@dataload
metadata about what was read and written, and this metadata is then used to track what
was read and written.
-This is our recommended first step when you're starting to use materialization in Hamilton.
+This is our recommended first step when you're starting to use materialization in Apache Hamilton.
Static materializers
~~~~~~~~~~~~~~~~~~~~
-Passing ``from_`` and ``to`` Hamilton objects to ``Builder().with_materializers()`` injects into the dataflow standardized nodes to load and save data. It solves the 3 limitations highlighted in the previous section:
+Passing ``from_`` and ``to`` Apache Hamilton objects to ``Builder().with_materializers()`` injects into the dataflow standardized nodes to load and save data. It solves the 3 limitations highlighted in the previous section:
-1. Redundancy β
: Using the ``from_`` and ``to`` Hamilton constructs reduces the boilerplate to load and save data from common formats (JSON, parquet, CSV, etc.) and to interact with 3rd party libraries (pandas, matplotlib, xgboost, dlt, etc.)
+1. Redundancy β
: Using the ``from_`` and ``to`` Apache Hamilton constructs reduces the boilerplate to load and save data from common formats (JSON, parquet, CSV, etc.) and to interact with 3rd party libraries (pandas, matplotlib, xgboost, dlt, etc.)
2. Observability β
: Loaders and savers are part of the dataflow. You can view them with ``Driver.display_all_functions()`` and execute nodes by requesting them with ``Driver.execute()``.
3. Flexibility β
: The loading and saving behavior is decoupled from the dataflow and can modified easily when creating the ``Driver`` and executing code.
@@ -102,7 +102,7 @@ The dataflow is executed by passing ``from_`` and ``to`` objects to ``Driver.mat
``Driver.materialize()`` can receive data savers (``from_``) and loaders (``to``) and will execute all ``to`` passed. Like ``Driver.execute()``, it can receive ``inputs``, and ``overrides``, but instead of ``final_vars`` it receives ``additional_vars``.
-1. Redundancy β
: Uses ``from_`` and ``to`` Hamilton constructs.
+1. Redundancy β
: Uses ``from_`` and ``to`` Apache Hamilton constructs.
2. Observability πΈ: Materializers are visible with ``Driver.visualize_materialization()``, but can't be introspected otherwise. Also, you need to rely on ``Driver.materialize()`` which has a different call signature.
3. Flexibility β
: Loading and saving is decoupled from the dataflow.
@@ -113,7 +113,7 @@ The dataflow is executed by passing ``from_`` and ``to`` objects to ``Driver.mat
Function modifiers
~~~~~~~~~~~~~~~~~~
-By adding ``@load_from`` and ``@save_to`` function modifiers (:ref:`loader-saver-decorators`) to Hamilton functions, materializers are generated when using ``Builder.with_modules()``. This approach ressembles **1) from Driver**:
+By adding ``@load_from`` and ``@save_to`` function modifiers (:ref:`loader-saver-decorators`) to Apache Hamilton functions, materializers are generated when using ``Builder.with_modules()``. This approach ressembles **1) from Driver**:
.. note::
@@ -131,7 +131,7 @@ By adding ``@load_from`` and ``@save_to`` function modifiers (:ref:`loader-saver
DataLoader and DataSaver
------------------------
-In Hamilton, ``DataLoader`` and ``DataSaver`` are classes that define how to load or save a particular data format. Calling ``Driver.materialize(DataLoader(), DataSaver())`` adds nodes to the dataflow (see visualizations above).
+In Apache Hamilton, ``DataLoader`` and ``DataSaver`` are classes that define how to load or save a particular data format. Calling ``Driver.materialize(DataLoader(), DataSaver())`` adds nodes to the dataflow (see visualizations above).
Here are simplified snippets for saving and loading an XGBoost model to/from JSON.
@@ -142,4 +142,4 @@ Here are simplified snippets for saving and loading an XGBoost model to/from JSO
| | |
+----------------------------------------------+-----------------------------------------------+
-To define your own DataSaver and DataLoader, the Hamilton `XGBoost extension <https://github.com/apache/hamilton/blob/main/hamilton/plugins/xgboost_extensions.py>`_ provides a good example
+To define your own DataSaver and DataLoader, the Apache Hamilton `XGBoost extension <https://github.com/apache/hamilton/blob/main/hamilton/plugins/xgboost_extensions.py>`_ provides a good example
diff --git a/docs/concepts/node.rst b/docs/concepts/node.rst
index 8fc6a6e7..e270a60b 100644
--- a/docs/concepts/node.rst
+++ b/docs/concepts/node.rst
@@ -2,22 +2,22 @@
Functions, nodes & dataflow
===========================
-On this page, you'll learn how Hamilton converts your Python functions into nodes and then creates a dataflow.
+On this page, you'll learn how Apache Hamilton converts your Python functions into nodes and then creates a dataflow.
Functions
---------
-Hamilton requires you to write your code using functions. To get started, you simply need to:
+Apache Hamilton requires you to write your code using functions. To get started, you simply need to:
- `Annotate the type <https://docs.python.org/3/library/typing.html>`_ of the function parameters and return value.
- Specify the function dependencies with the parameter names.
- Store your code in Python modules (``.py`` files).
-Since your code doesn't depend on special "Hamilton code", you can reuse it however you want!
+Since your code doesn't depend on special "Apache Hamilton code", you can reuse it however you want!
Specifying dependencies
~~~~~~~~~~~~~~~~~~~~~~~
-In Hamilton, you define dependencies by matching parameter names with the names of other functions. Below, the function name and return type ``A() -> int`` match the parameter ``A: int`` found in functions ``B()`` and ``C()``.
+In Apache Hamilton, you define dependencies by matching parameter names with the names of other functions. Below, the function name and return type ``A() -> int`` match the parameter ``A: int`` found in functions ``B()`` and ``C()``.
.. code-block:: python
@@ -37,7 +37,7 @@ In Hamilton, you define dependencies by matching parameter names with the names
.. image:: ../_static/abc_basic.png
:align: center
-The figure shows how Hamilton automatically assembled the functions ``A()``, ``B()``, and ``C()``.
+The figure shows how Apache Hamilton automatically assembled the functions ``A()``, ``B()``, and ``C()``.
Helper function
~~~~~~~~~~~~~~~~
@@ -62,20 +62,20 @@ You can prefix a function name with an underscore (``_``) to prevent it from bei
Function naming tips
~~~~~~~~~~~~~~~~~~~~
-Hamilton strongly agrees with the `Zen of Python <https://peps.python.org/pep-0020/>`_ #2: "Explicit is better than implicit". Meaningful function names help document what functions do, so don't shy away from longer names. If you were to come across a function named ``life_time_value`` versus ``ltv`` versus ``l_t_v``, which one is most obvious? Remember your code usually lives a lot longer than you ever think it will.
+Apache Hamilton strongly agrees with the `Zen of Python <https://peps.python.org/pep-0020/>`_ #2: "Explicit is better than implicit". Meaningful function names help document what functions do, so don't shy away from longer names. If you were to come across a function named ``life_time_value`` versus ``ltv`` versus ``l_t_v``, which one is most obvious? Remember your code usually lives a lot longer than you ever think it will.
-Unlike the common practice of including meaningful verbs in function names (e.g., ``get_credentials()``, ``statistical_test()``), with Hamilton, the function name should more closely align with nouns. That's because the function name determines the node name and how data will be queried. Therefore, names that describe the node result rather than its action may be more readable (e.g., ``credentials()``, ``statistical_results()``).
+Unlike the common practice of including meaningful verbs in function names (e.g., ``get_credentials()``, ``statistical_test()``), with Apache Hamilton, the function name should more closely align with nouns. That's because the function name determines the node name and how data will be queried. Therefore, names that describe the node result rather than its action may be more readable (e.g., ``credentials()``, ``statistical_results()``).
Nodes
-----
-A node is a single "operation" or "step" in a dataflow. Hamilton users write Python `functions` that Hamilton converts into `nodes`. User never directly create nodes.
+A node is a single "operation" or "step" in a dataflow. Apache Hamilton users write Python `functions` that Apache Hamilton converts into `nodes`. User never directly create nodes.
Anatomy of a node
~~~~~~~~~~~~~~~~~
-The following figure and table detail how a Python function maps to a Hamilton node.
+The following figure and table detail how a Python function maps to a Apache Hamilton node.
.. image:: ../_static/function_anatomy.png
@@ -108,9 +108,9 @@ Since functions almost always map to nodes 1-to-1, the two terms are often used
Dataflow
--------
-From a collection of nodes, Hamilton automatically assembles the dataflow. For each node, it creates edges between itself and its dependencies, resulting in a `dataflow <https://en.wikipedia.org/wiki/Dataflow_programming>`_ (or a `graph <https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)>`_ in more mathematical terms).
+From a collection of nodes, Apache Hamilton automatically assembles the dataflow. For each node, it creates edges between itself and its dependencies, resulting in a `dataflow <https://en.wikipedia.org/wiki/Dataflow_programming>`_ (or a `graph <https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)>`_ in more mathematical terms).
-From the user perspective, you give Hamilton a Python module containing your functions and it will generate your dataflow! This is a key difference with popular orchestration / pipeline / workflow frameworks (Airflow, Kedro, Prefect, VertexAI, SageMaker, etc.)
+From the user perspective, you give Apache Hamilton a Python module containing your functions and it will generate your dataflow! This is a key difference with popular orchestration / pipeline / workflow frameworks (Airflow, Kedro, Prefect, VertexAI, SageMaker, etc.)
How other frameworks build graphs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -118,23 +118,23 @@ In most frameworks, you first define nodes / steps / tasks / components. Then, y
Readability
^^^^^^^^^^^
-In that case, the code for ``step A`` doesn't tell you how it relates ``step B`` or the broader dataflow. Hamilton solves this problem by tying functions, nodes, and dataflow definitions in a single place. The ratio of reading to writing code can be as high as `10:1 <https://www.goodreads.com/quotes/835238-indeed-the-ratio-of-time-spent-reading-versus-writing-is>`_, especially for complex dataflows, so optimizing for readability is high-value.
+In that case, the code for ``step A`` doesn't tell you how it relates ``step B`` or the broader dataflow. Apache Hamilton solves this problem by tying functions, nodes, and dataflow definitions in a single place. The ratio of reading to writing code can be as high as `10:1 <https://www.goodreads.com/quotes/835238-indeed-the-ratio-of-time-spent-reading-versus-writing-is>`_, especially for complex dataflows, so optimizing for readability is high-value.
Maintainability
^^^^^^^^^^^^^^^
-Typically, editing a dataflow (new feature, debugging, etc.) alters both what a **node** does and how the **dataflow** is structured. Consequently, changes to ``step A`` require you to manually ensure consistent edits to the definition of dataflows, which is likely in another file. In enterprise settings, it can become difficult to discover and track every location where ``step A`` is used (potentially 10s or 100s of pipelines), increasing the likelihood of breaking changes. Hamilton avo [...]
+Typically, editing a dataflow (new feature, debugging, etc.) alters both what a **node** does and how the **dataflow** is structured. Consequently, changes to ``step A`` require you to manually ensure consistent edits to the definition of dataflows, which is likely in another file. In enterprise settings, it can become difficult to discover and track every location where ``step A`` is used (potentially 10s or 100s of pipelines), increasing the likelihood of breaking changes. Apache Hamil [...]
Recap
--------
- Users write Python functions into modules with proper naming and typing
- Helper functions use an underscore prefix (e.g., ``_helper()``)
-- Hamilton converts functions into nodes
-- Hamilton automatically assembles nodes into a dataflow
+- Apache Hamilton converts functions into nodes
+- Apache Hamilton automatically assembles nodes into a dataflow
Next step
---------
-So far, we learned how to write Hamilton code for our dataflow. Next, we'll explore how we can effectively
+So far, we learned how to write Apache Hamilton code for our dataflow. Next, we'll explore how we can effectively
1. Convert a Python module into dataflow
2. Visualize a dataflow
diff --git a/docs/concepts/parallel-task.rst b/docs/concepts/parallel-task.rst
index a5895fef..e5bdfa11 100644
--- a/docs/concepts/parallel-task.rst
+++ b/docs/concepts/parallel-task.rst
@@ -1,7 +1,7 @@
Dynamic DAGs/Parallel Execution
----------------------------------
-There are two approaches to parallel execution in Hamilton:
+There are two approaches to parallel execution in Apache Hamilton:
1. Using an adapter that submits each node/function to a system that handles execution, e.g. ray, dask, async, or a threadpool.
2. Using the `Parallelizable[]` and `Collect[]` types + delegating to an executor.
@@ -9,10 +9,10 @@ There are two approaches to parallel execution in Hamilton:
Using an Adapter
================
The adapter approach effectively farms out the execution of each node/function to a system that can handle resolving
-futures. That is, Hamilton walks the DAG and submits each node to the adapter, which then submits the node for execution,
+futures. That is, Apache Hamilton walks the DAG and submits each node to the adapter, which then submits the node for execution,
and internally the execution resolves any Futures from prior submitted nodes.
-To make use of this, the general pattern is you apply an adapter to the driver and don't need to touch your Hamilton functions!:
+To make use of this, the general pattern is you apply an adapter to the driver and don't need to touch your Apache Hamilton functions!:
.. code-block:: python
@@ -43,7 +43,7 @@ Using the `Parallelizable[]` and `Collect[]` types
==================================================
-Hamilton now has pluggable execution, which allows for the following:
+Apache Hamilton now has pluggable execution, which allows for the following:
1. Grouping of nodes into "tasks" (discrete execution unit between serialization boundaries)
2. Executing the tasks in parallel, using any executor of your choice
@@ -132,7 +132,7 @@ Challenge:
Solution:
* Make sure that your objects are serializable.
-* If you're using a library that doesn't support serialization, then one option is to have Hamilton instantiate
+* If you're using a library that doesn't support serialization, then one option is to have Apache Hamilton instantiate
the object in each parallel block. You can do this by making the code depend on something within the parallel block.
* Another option is write a customer wrapper function that uses `__set_state__` and `__get_state__` to serialize and deserialize the object.
* See `this issue <https://github.com/apache/hamilton/issues/743>`_ for details and possible features to make
diff --git a/docs/concepts/visualization.rst b/docs/concepts/visualization.rst
index 9f027719..1e114384 100644
--- a/docs/concepts/visualization.rst
+++ b/docs/concepts/visualization.rst
@@ -2,7 +2,7 @@
Visualization
=============
-After assembling the dataflow, several visualization features become available to the Driver. Hamilton dataflow visualizations are great for documentation because they are directly derived from the code.
+After assembling the dataflow, several visualization features become available to the Driver. Apache Hamilton dataflow visualizations are great for documentation because they are directly derived from the code.
On this page, you'll learn:
@@ -124,7 +124,7 @@ Learn more about :doc:`materialization`.
View node dependencies
----------------------
-Representing data pipelines, ML experiments, or LLM applications as a dataflow helps reason about the dependencies between operations. The Hamilton Driver has the following utilities to select and return a list of nodes (to learn more :doc:`../how-tos/use-hamilton-for-lineage`):
+Representing data pipelines, ML experiments, or LLM applications as a dataflow helps reason about the dependencies between operations. The Apache Hamilton Driver has the following utilities to select and return a list of nodes (to learn more :doc:`../how-tos/use-hamilton-for-lineage`):
- ``.what_is_upstream_of(*node_names: str)``
- ``.what_is_downstream_of(*node_names: str)``
@@ -193,7 +193,7 @@ This `online graphviz editor <https://edotor.net/>`_ can help you get started!
) -> Tuple[dict, Optional[str], Optional[str]]:
"""Custom style function for the visualization.
- :param node: node that Hamilton is styling.
+ :param node: node that Apache Hamilton is styling.
:param node_class: class used to style the default visualization
:return: a triple of (style, node_class, legend_name)
"""
diff --git a/docs/get-started/index.rst b/docs/get-started/index.rst
index aa68bc01..360d5ccc 100644
--- a/docs/get-started/index.rst
+++ b/docs/get-started/index.rst
@@ -2,25 +2,25 @@
Get Started
====================
-Welcome to Hamilton's documentation!
+Welcome to Apache Hamilton's documentation!
www.tryhamilton.dev
--------------------
-Before diving in, we highly recommend you try Hamilton in your browser at https://www.tryhamilton.dev.
+Before diving in, we highly recommend you try Apache Hamilton in your browser at https://www.tryhamilton.dev.
It allows you to:
- 1. run python in the browser, so you can get a feel for the basics of Hamilton *without installing anything!*
+ 1. run python in the browser, so you can get a feel for the basics of Apache Hamilton *without installing anything!*
2. it includes various examples that you can run and modify.
- 3. it represents an easy hands-on introduction to Hamilton that should get you comfortable with the framework and \
+ 3. it represents an easy hands-on introduction to Apache Hamilton that should get you comfortable with the framework and \
its basic capabilities.
-Get started with Hamilton locally
+Get started with Apache Hamilton locally
-------------------------------------
-The following section of the docs will teach you how to install Hamilton and get started with your own project.
+The following section of the docs will teach you how to install Apache Hamilton and get started with your own project.
.. toctree::
diff --git a/docs/get-started/install.md b/docs/get-started/install.md
index 2e630c69..c06e8b2d 100644
--- a/docs/get-started/install.md
+++ b/docs/get-started/install.md
@@ -3,7 +3,7 @@ Installing hamilton is easy!
# Install
-Hamilton is a lightweight framework with a variety of extensions/plugins. To get started, you'll need the following:
+Apache Hamilton is a lightweight framework with a variety of extensions/plugins. To get started, you'll need the following:
- ``python >= 3.8``
- ``pip``
@@ -12,7 +12,7 @@ For help with python/pip/managing virtual environments see the [python docs](htt
## Installing with pip
-Hamilton is published on [pypi](https://pypi.org/project/sf-hamilton/) under ``sf-hamilton``. To install, run:
+Apache Hamilton is published on [pypi](https://pypi.org/project/sf-hamilton/) under ``sf-hamilton``. To install, run:
`pip install sf-hamilton`
@@ -26,7 +26,7 @@ operating system.*
## Installing with conda
-Hamilton is also available on conda if you prefer:
+Apache Hamilton is also available on conda if you prefer:
`conda install -c hamilton-opensource sf-hamilton`
diff --git a/docs/get-started/learning-resources.md b/docs/get-started/learning-resources.md
index 93e310a2..15f20baf 100644
--- a/docs/get-started/learning-resources.md
+++ b/docs/get-started/learning-resources.md
@@ -1,26 +1,26 @@
# Learning Resources
-Several channels are available to get started with Hamilton, learn advanced usage, and participate in the latest feature development.
+Several channels are available to get started with Apache Hamilton, learn advanced usage, and participate in the latest feature development.
## π User Guide Documentation
-The [user guide](../concepts/index.rst) gives a complete overview of Hamilton's features.
+The [user guide](../concepts/index.rst) gives a complete overview of Apache Hamilton's features.
## π Reference Documentation
-The [reference documentation](../reference/dataflows/index.rst) details Hamilton's public API.
+The [reference documentation](../reference/dataflows/index.rst) details Apache Hamilton's public API.
## β tryhamilton.dev
-The [tryhamilton.dev](https://tryhamilton.dev) website provides interactive tutorials in-browser to learn specific Hamilton concepts.
+The [tryhamilton.dev](https://tryhamilton.dev) website provides interactive tutorials in-browser to learn specific Apache Hamilton concepts.
## π Dataflow Hub
-The [Hamilton Dataflow Hub](https://hub.dagworks.io/docs/) hosts user-created dataflows that are easy to download and reuse in your project.
+The [Apache Hamilton Dataflow Hub](https://hub.dagworks.io/docs/) hosts user-created dataflows that are easy to download and reuse in your project.
## π‘ Blog
-The [DAGWorks Blog](https://blog.dagworks.io/) publishes articles on problems Hamilton helps solve, reference architectures, and new features.
+The [DAGWorks Blog](https://blog.dagworks.io/) publishes articles on problems Apache Hamilton helps solve, reference architectures, and new features.
## π Slack
The [Slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g) is the ideal place to ask questions, request features, and give feedback.
@@ -28,78 +28,78 @@ The [Slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/z
## π£ Talks
See [our youtube for the most up to date recordings](https://www.youtube.com/@DAGWorks-Inc/playlists) - we are slow to list them here.
-* 2024-02 Hamilton Meet-up for February
+* 2024-02 Apache Hamilton Meet-up for February
* [Recording](https://www.youtube.com/watch?v=ks672Lm0CJo.)
- * [Slides](https://github.com/skrawcz/talks/files/14351139/Hamilton.February.2024.Meetup.pdf)
+ * [Slides](https://github.com/skrawcz/talks/files/14351139/Apache Hamilton.February.2024.Meetup.pdf)
-* 2023-12 Why you should build your GenAI/LLM apps using Hamilton. [AICamp End of Year in SF](https://www.aicamp.ai/event/eventdetails/W2023121217)
+* 2023-12 Why you should build your GenAI/LLM apps using Apache Hamilton. [AICamp End of Year in SF](https://www.aicamp.ai/event/eventdetails/W2023121217)
* [Recording](https://youtu.be/IwWixrjhkZU?si=DVa72Zr4iD-hibS5&t=7602)
- * [Slides](https://github.com/skrawcz/talks/files/13666470/Why.you.should.build.your.GenAI_LLM.apps.using.Hamilton.pdf)
+ * [Slides](https://github.com/skrawcz/talks/files/13666470/Why.you.should.build.your.GenAI_LLM.apps.using.Apache Hamilton.pdf)
* 2023-12 Bridging Classic ML Pipelines with the World of LLMs. [PyData Global](https://global2023.pydata.org/cfp/talk/3REDA9/)
* [Slides](https://github.com/skrawcz/talks/files/13666479/Bridging.Classic.ML.Pipelines.with.the.World.of.LLMs.1.pdf)
-* 2023-11 Hamilton: Natively bringing software engineering best practices to python data transformations. [Scale by the Bay](https://www.scale.bythebay.io/).
+* 2023-11 Apache Hamilton: Natively bringing software engineering best practices to python data transformations. [Scale by the Bay](https://www.scale.bythebay.io/).
* [Recording](https://www.youtube.com/watch?v=gK4-6X0h7PU)
* [Slides](https://github.com/skrawcz/talks/files/13969784/Scale.By.The.Bay.-.Hamilton_.Natively.bringing.SWE.best.practices.to.python.data.transformations.pdf)
-* 2023-09 Hamilton: Natively bringing software engineering best practices to python data transformations. [Bay Area Python Interest Group (BAYPIGgies)](https://www.meetup.com/baypiggies/events/296283989/)
- * [Slides](https://github.com/skrawcz/talks/files/12785978/BayPIGgies_.Hamilton.Talk.pdf)
+* 2023-09 Apache Hamilton: Natively bringing software engineering best practices to python data transformations. [Bay Area Python Interest Group (BAYPIGgies)](https://www.meetup.com/baypiggies/events/296283989/)
+ * [Slides](https://github.com/skrawcz/talks/files/12785978/BayPIGgies_.Apache Hamilton.Talk.pdf)
-* 2023-08 dbt + Hamilton: Enabling you to maintain complex Python within dbt models. [MDSFest'23](https://www.mdsfest.com/)
+* 2023-08 dbt + Apache Hamilton: Enabling you to maintain complex Python within dbt models. [MDSFest'23](https://www.mdsfest.com/)
* [Recording](https://www.youtube.com/watch?v=ZM-kM8DqlaQ&list=PLdVpUmZrh0QpDi07ENp3FD5aTFuTTtWnP)
* [Slides](https://github.com/skrawcz/talks/files/12431755/dbt.%2B.Hamilton_.Enabling.you.to.maintain.complex.python.within.dbt.models.pdf)
-* 2023-06 Hamilton: an OS tool to add to your LLM App toolbelt. LLM Avalanche.
+* 2023-06 Apache Hamilton: an OS tool to add to your LLM App toolbelt. LLM Avalanche.
* [Slides](https://github.com/skrawcz/talks/files/11899349/Hamilton_.an.OS.tool.to.add.to.your.LLM.App.toolbelt.pdf)
-* 2023-06 Feature Engineering with Hamilton: Portability & Lineage. [Budapest ML Forum June 2023](https://budapestml.hu/2023/en/)
+* 2023-06 Feature Engineering with Apache Hamilton: Portability & Lineage. [Budapest ML Forum June 2023](https://budapestml.hu/2023/en/)
* [Slides](https://github.com/skrawcz/talks/files/11690901/Stefan_Krawczyk_BudapestTalkJune2023_FeatureEngineeringwith.Hamilton_Portability.Lineage.pdf)
* 2023-06 British Cycling Data Platform in Python. Manchester PyData Meetup
* [Slides](https://github.com/skrawcz/talks/files/11899331/PyData.British.Cycling.7.June.2023.pdf)
* Co-presented with Peter Robinson, and Murray Tait.
-* 2023-04 Lightweight Lineage with Hamilton. PyData Seattle
- * [Slides](https://github.com/skrawcz/talks/files/11399972/PyData-Seattl-Lightning-Talk-2023-Lighweight-Lineage-with-Hamilton.pdf)
+* 2023-04 Lightweight Lineage with Apache Hamilton. PyData Seattle
+ * [Slides](https://github.com/skrawcz/talks/files/11399972/PyData-Seattl-Lightning-Talk-2023-Lighweight-Lineage-with-Apache Hamilton.pdf)
-* 2023-01 Hamilton: Natively bringing software engineering best practices to python data transformations. AI Camp Meetup San Jose
+* 2023-01 Apache Hamilton: Natively bringing software engineering best practices to python data transformations. AI Camp Meetup San Jose
* [Slides](https://github.com/skrawcz/talks/files/10830349/Hamilton_.Natively.bringing.software.engineering.best.practices.to.python.data.transformations.-.January.2023.pdf)
-* 2022-10 Hamilton: an open source, declarative, micro-framework for clean & robust feature transform code in Python. Feature Store Summit
+* 2022-10 Apache Hamilton: an open source, declarative, micro-framework for clean & robust feature transform code in Python. Feature Store Summit
* [Event](https://www.featurestoresummit.com/)
- * [Slides](https://github.com/skrawcz/talks/files/9759661/FS.Summit.2022.-.Hamilton.pdf)
+ * [Slides](https://github.com/skrawcz/talks/files/9759661/FS.Summit.2022.-.Apache Hamilton.pdf)
-* 2022-09 Hamilton: enabling software engineering best practices for data transformations via generalized dataflow graphs. DEco - First International Workshop on Data Ecosystems
+* 2022-09 Apache Hamilton: enabling software engineering best practices for data transformations via generalized dataflow graphs. DEco - First International Workshop on Data Ecosystems
* [Event](https://dbis.rwth-aachen.de/DEco22/)
* [Slides](https://github.com/skrawcz/talks/files/9550914/Submitted.-.DEco.2022_.Hamilton_.enabling.software.engineering.best.practices.for.data.transformations.via.generalized.dataflow.graphs.1.pdf)
-* 2022-09 Hamilton: a modular open source declarative paradigm for high level modeling of dataflows. CDMS - First International Workshop on Composable Data Management Systems
+* 2022-09 Apache Hamilton: a modular open source declarative paradigm for high level modeling of dataflows. CDMS - First International Workshop on Composable Data Management Systems
* [Event](https://cdmsworkshop.github.io/2022/)
* [Slides](https://github.com/skrawcz/talks/files/9550939/CDMS.2022.-.Hamilton_.a.modular.open.source.declarative.paradigm.for.high.level.modeling.of.dataflows.1.pdf)
* [Paper](https://cdmsworkshop.github.io/2022/Proceedings/ShortPapers/Paper6_StefanKrawczyk.pdf)
-* 2022-08 Hamilton: A Python Micro-Framework for tidy scalable Pandas. Scalable Pandas Meetup
+* 2022-08 Apache Hamilton: A Python Micro-Framework for tidy scalable Pandas. Scalable Pandas Meetup
* [Recording](https://www.youtube.com/watch?v=m_rjCzxQj4c&ab_channel=Ponder)
- * [Slides](https://github.com/skrawcz/talks/files/9428705/Hamilton.%40.Ponder.Pandas.meetup.pdf)
+ * [Slides](https://github.com/skrawcz/talks/files/9428705/Apache Hamilton.%40.Ponder.Pandas.meetup.pdf)
-* 2022-08 Scalable feature engineering with Hamilton on Ray. Ray Summit
- * [Slides](https://github.com/skrawcz/talks/files/9411082/Submitted.Slides.-.Ray.Summit_.Scalable.feature.engineering.with.Hamilton.on.Ray.pdf)
+* 2022-08 Scalable feature engineering with Apache Hamilton on Ray. Ray Summit
+ * [Slides](https://github.com/skrawcz/talks/files/9411082/Submitted.Slides.-.Ray.Summit_.Scalable.feature.engineering.with.Apache Hamilton.on.Ray.pdf)
-* 2022-07 Hamilton: A Python Micro-Framework for Data / Feature Engineering. MLOPsWorld Bay Area
+* 2022-07 Apache Hamilton: A Python Micro-Framework for Data / Feature Engineering. MLOPsWorld Bay Area
* [Slides](https://github.com/skrawcz/talks/files/9213924/Hamilton_.A.Python.Micro-Framework.for.Data._.Feature.Engineering.pdf)
-* 2022-05 Hamilton: a python micro-framework for data / feature engineering at Stitch Fix. AICamp
+* 2022-05 Apache Hamilton: a python micro-framework for data / feature engineering at Stitch Fix. AICamp
* [Recording](https://www.youtube.com/watch?v=PDGIt37dov8)
- * [Slides](https://github.com/skrawcz/talks/files/8691633/AICamp.Hamilton.Presentation.pdf)
+ * [Slides](https://github.com/skrawcz/talks/files/8691633/AICamp.Apache Hamilton.Presentation.pdf)
-* 2022-02 [Open Source] Hamilton, a micro framework for creating dataframes, and its application at Stitch Fix. Apply(Meetup)
+* 2022-02 [Open Source] Apache Hamilton, a micro framework for creating dataframes, and its application at Stitch Fix. Apply(Meetup)
* [Event](https://www.applyconf.com/agenda/open-source-hamilton-a-micro-framework-for-creating-dataframes-and-its-application-at-stitch-fix/).
* [Recording](https://www.youtube.com/watch?v=CHfrT5OVjlM)
* [Slides](https://github.com/skrawcz/talks/blob/main/Public%20ApplyConf2022%20-%20%5BOpen%20Source%5D%20Hamilton%2C%20a%20micro%20framework%20for%20creating%20dataframes%2C%20and%20its%20application%20at%20Stitch%20Fix.pdf)
-* 2021-12 Hamilton an open source micro framework for creating dataframes. SF Python Meetup
+* 2021-12 Apache Hamilton an open source micro framework for creating dataframes. SF Python Meetup
* [Recording](https://www.youtube.com/watch?v=_XUYfwougz4)
* [Slides](https://github.com/skrawcz/talks/files/8944605/Python.Meetup.Dec.2021.-.Hamilton_.an.open.source.micro.framework.for.creating.dataframes.pdf)
@@ -107,37 +107,37 @@ See [our youtube for the most up to date recordings](https://www.youtube.com/@DA
## π° External Blogs
For the latest blog posts, see the [DAGWorks's Blog](https://blog.dagworks.io/).
-* 2024-03 [RAG: ingestion and chunking using Hamilton and scaling to Ray, Dask, or PySpark](https://blog.dagworks.io/p/rag-ingestion-and-chunking-using)
+* 2024-03 [RAG: ingestion and chunking using Apache Hamilton and scaling to Ray, Dask, or PySpark](https://blog.dagworks.io/p/rag-ingestion-and-chunking-using)
* 2024-02 [A command line tool to improve your development workflow](https://blog.dagworks.io/p/a-command-line-tool-to-improve-your)
* 2024-02 [Monthly Meetup Recap and office hours](https://blog.dagworks.io/p/monthly-hamilton-meetup-and-office)
* 2024-02 [Using IPython Jupyter Magic commands to improve the notebook experience](https://blog.dagworks.io/p/using-ipython-jupyter-magic-commands)
* 2024-02 [Building a lightweight experiment manager](https://blog.dagworks.io/p/building-a-lightweight-experiment)
-* 2024-01 [Customizing Hamiltonβs Execution with the new Lifecycle API](https://blog.dagworks.io/p/customizing-hamiltons-execution-with)
+* 2024-01 [Customizing Apache Hamiltonβs Execution with the new Lifecycle API](https://blog.dagworks.io/p/customizing-hamiltons-execution-with)
* 2024-01 [How well-structured should your data code be?](https://blog.dagworks.io/p/how-well-structured-should-your-data)
* 2024-01 [From Dev to Prod: a ML Pipeline Reference Post](https://blog.dagworks.io/p/from-dev-to-prod-a-ml-pipeline-reference)
* 2023-12 [Winning over hearts and minds at work: ADKAR my favorite change management approach](https://blog.dagworks.io/p/winning-hearts-and-minds-at-work)
-* 2023-11 [π Weβre launching the Hamilton Dataflow Hub!](https://blog.dagworks.io/p/were-launching-the-hamilton-dataflow)
+* 2023-11 [π Weβre launching the Apache Hamilton Dataflow Hub!](https://blog.dagworks.io/p/were-launching-the-hamilton-dataflow)
* 2023-10 [Separate data I/O from transformation -- your future self will thank you.](https://blog.dagworks.io/p/separate-data-io-from-transformation)
-* 2023-09 [Retrieval augmented generation (RAG) with Streamlit, FastAPI, Weaviate, and Hamilton!](https://blog.dagworks.io/p/retrieval-augmented-generation-reference-arch)
-* 2023-09 [LLMOps: Production prompt engineering patterns with Hamilton](https://blog.dagworks.io/p/llmops-production-prompt-engineering)
-* 2023-09 [Feature Engineering with Hamilton](https://blog.dagworks.io/p/feature-engineering-with-hamilton)
-* 2023-08 [Expressing PySpark Transformations Declaratively with Hamilton](https://blog.dagworks.io/p/expressing-pyspark-transformations)
-* 2023-08 [Containerized PDF Summarizer with FastAPI and Hamilton](https://blog.dagworks.io/p/containerized-pdf-summarizer-with)
-* 2023-08 [Dynamic DAGs: Counting Stars with Hamilton](https://blog.dagworks.io/p/counting-stars-with-hamilton)
-* 2023-08 [Featurization: Integrating Hamilton with Feast](https://blog.dagworks.io/p/featurization-integrating-hamilton)
-* 2023-07 [Simplify Prefect Workflow Creation and Maintenance with Hamilton](https://blog.dagworks.io/p/simplify-prefect-workflow-creation)
-* 2023-07 [Building a maintainable and modular LLM application stack with Hamilton](https://blog.dagworks.io/p/building-a-maintainable-and-modular)
-* 2023-06 [Simplify Airflow DAG Creation and Maintenance with Hamilton](https://blog.dagworks.io/p/supercharge-your-airflow-dag-with)
-* 2023-05 [Lineage + Hamilton in 10 minutes](https://blog.dagworks.io/p/lineage-hamilton-in-10-minutes-c2b8a944e2e6)
-* 2022-11 [Hamilton + DBT in 5 minutes](https://blog.dagworks.io/p/hamilton-dbt-in-5-minutes-62e4cb63f08f)
-* 2022-07 [Tidy production pandas with Hamilton](https://towardsdatascience.com/tidy-production-pandas-with-hamilton-3b759a2bf562)
-* 2022-06 [Developing Scalable Feature Engineering DAGs with Metaflow & Hamilton](https://outerbounds.com/blog/developing-scalable-feature-engineering-dags)
-* 2022-05 [Hamilton backstory and intro post on TDS](https://towardsdatascience.com/functions-dags-introducing-hamilton-a-microframework-for-dataframe-generation-more-8e34b84efc1d)
-* 2022-05 [Hamilton + Pandas in five minutes](https://towardsdatascience.com/how-to-use-hamilton-with-pandas-in-5-minutes-89f63e5af8f5)
-* 2022-05 [Iterating with Hamilton in a Notebook](https://towardsdatascience.com/how-to-use-hamilton-with-pandas-in-5-minutes-89f63e5af8f5)
+* 2023-09 [Retrieval augmented generation (RAG) with Streamlit, FastAPI, Weaviate, and Apache Hamilton!](https://blog.dagworks.io/p/retrieval-augmented-generation-reference-arch)
+* 2023-09 [LLMOps: Production prompt engineering patterns with Apache Hamilton](https://blog.dagworks.io/p/llmops-production-prompt-engineering)
+* 2023-09 [Feature Engineering with Apache Hamilton](https://blog.dagworks.io/p/feature-engineering-with-hamilton)
+* 2023-08 [Expressing PySpark Transformations Declaratively with Apache Hamilton](https://blog.dagworks.io/p/expressing-pyspark-transformations)
+* 2023-08 [Containerized PDF Summarizer with FastAPI and Apache Hamilton](https://blog.dagworks.io/p/containerized-pdf-summarizer-with)
+* 2023-08 [Dynamic DAGs: Counting Stars with Apache Hamilton](https://blog.dagworks.io/p/counting-stars-with-hamilton)
+* 2023-08 [Featurization: Integrating Apache Hamilton with Feast](https://blog.dagworks.io/p/featurization-integrating-hamilton)
+* 2023-07 [Simplify Prefect Workflow Creation and Maintenance with Apache Hamilton](https://blog.dagworks.io/p/simplify-prefect-workflow-creation)
+* 2023-07 [Building a maintainable and modular LLM application stack with Apache Hamilton](https://blog.dagworks.io/p/building-a-maintainable-and-modular)
+* 2023-06 [Simplify Airflow DAG Creation and Maintenance with Apache Hamilton](https://blog.dagworks.io/p/supercharge-your-airflow-dag-with)
+* 2023-05 [Lineage + Apache Hamilton in 10 minutes](https://blog.dagworks.io/p/lineage-hamilton-in-10-minutes-c2b8a944e2e6)
+* 2022-11 [Apache Hamilton + DBT in 5 minutes](https://blog.dagworks.io/p/hamilton-dbt-in-5-minutes-62e4cb63f08f)
+* 2022-07 [Tidy production pandas with Apache Hamilton](https://towardsdatascience.com/tidy-production-pandas-with-hamilton-3b759a2bf562)
+* 2022-06 [Developing Scalable Feature Engineering DAGs with Metaflow & Apache Hamilton](https://outerbounds.com/blog/developing-scalable-feature-engineering-dags)
+* 2022-05 [Apache Hamilton backstory and intro post on TDS](https://towardsdatascience.com/functions-dags-introducing-hamilton-a-microframework-for-dataframe-generation-more-8e34b84efc1d)
+* 2022-05 [Apache Hamilton + Pandas in five minutes](https://towardsdatascience.com/how-to-use-hamilton-with-pandas-in-5-minutes-89f63e5af8f5)
+* 2022-05 [Iterating with Apache Hamilton in a Notebook](https://towardsdatascience.com/how-to-use-hamilton-with-pandas-in-5-minutes-89f63e5af8f5)
## π Podcasts
-* 2024-03 [Hamilton mention in Real Python, about ipython magic command post](https://realpython.com/podcasts/rpp/196/)
+* 2024-03 [Apache Hamilton mention in Real Python, about ipython magic command post](https://realpython.com/podcasts/rpp/196/)
* 2023-06 [Exploring the Intersection of DAGs, ML Code, and Complex Code Bases: An Elegant Solution Unveiled with Stefan Krawczyk of DAGWorks](https://datastackshow.com/podcast/exploring-the-intersection-of-dags-ml-code-and-complex-code-bases-an-elegant-solution-unveiled-with-stefan-krawczyk-of-dagworks/)
* 2022-08 [S01 E08 - MLOps Week 8: The MLOps Mindset with Stefan Krawczyk](https://rss.com/podcasts/mlops-weekly/571949/)
* 2022-04 [MLOps dla 100 data scientistΓ³w](https://nieliniowy.pl/mlops-dla-100-data-scientistow-stefan-krawczyk-stitch-fix/) (in Polish)
diff --git a/docs/get-started/license.rst b/docs/get-started/license.rst
index b7f1ed48..71576cc6 100644
--- a/docs/get-started/license.rst
+++ b/docs/get-started/license.rst
@@ -2,13 +2,13 @@
License
=======
-Hamilton is released under the `Apache 2.0 License <https://github.com/apache/hamilton/blob/main/LICENSE>`_.
+Apache Hamilton is released under the `Apache 2.0 License <https://github.com/apache/hamilton/blob/main/LICENSE>`_.
Usage analytics & data privacy
-----------------------------------
-By default, when using Hamilton, it collects anonymous usage data to help improve Hamilton and know where to apply development
+By default, when using Apache Hamilton, it collects anonymous usage data to help improve Apache Hamilton and know where to apply development
efforts.
We capture three types of events: one when the `Driver` object is instantiated, one when the `execute()` call on the \
@@ -17,9 +17,9 @@ No user data or potentially sensitive information is or ever will be collected.
* Operating System and Python version
* A persistent UUID to indentify the session, stored in ~/.hamilton.conf.
-* Error stack trace limited to Hamilton code, if one occurs.
-* Information on what features you're using from Hamilton: decorators, adapters, result builders.
-* How Hamilton is being used: number of final nodes in DAG, number of modules, size of objects passed to `execute()`, \
+* Error stack trace limited to Apache Hamilton code, if one occurs.
+* Information on what features you're using from Apache Hamilton: decorators, adapters, result builders.
+* How Apache Hamilton is being used: number of final nodes in DAG, number of modules, size of objects passed to `execute()`, \
the name of the Driver function being invoked.
diff --git a/docs/get-started/why-hamilton.md b/docs/get-started/why-hamilton.md
index d000ea8c..b477fcb6 100644
--- a/docs/get-started/why-hamilton.md
+++ b/docs/get-started/why-hamilton.md
@@ -1,12 +1,12 @@
-# Why use Hamilton?
+# Why use Apache Hamilton?
There are many choices for building dataflows/pipelines/workflows/ETLs.
-Let's compare Hamilton to some of the other options to help answer this question.
+Let's compare Apache Hamilton to some of the other options to help answer this question.
## Comparison to Other Frameworks
There are a lot of frameworks out there, especially in the pipeline space. This section should help you figure out when to
-use Hamilton with another framework, or in place of a framework, or when to use another framework altogether.
+use Apache Hamilton with another framework, or in place of a framework, or when to use another framework altogether.
Let's go over some groups of "competitive" or "complimentary" products. For a basic overview,
see the product matrix on the [homepage](../index.md).
@@ -18,16 +18,16 @@ Examples include:
- [Luigi](https://github.com/spotify/luigi)
- [dbt](https://www.getdbt.com/)
-Hamilton is not, in itself a macro, i.e. high level, task orchestration system. While it does orchestrate functions,
+Apache Hamilton is not, in itself a macro, i.e. high level, task orchestration system. While it does orchestrate functions,
and the DAG abstraction is very powerful, it does not provision compute,
-or schedule long-running jobs. Hamilton works well in conjunction with these macro systems.
-Hamilton provides the capabilities of fine-grained lineage, highly readable code, and self-documenting pipelines,
+or schedule long-running jobs. Apache Hamilton works well in conjunction with these macro systems.
+Apache Hamilton provides the capabilities of fine-grained lineage, highly readable code, and self-documenting pipelines,
which many of these systems lack.
-Hamilton can be used within any python orchestration system in the following ways:
+Apache Hamilton can be used within any python orchestration system in the following ways:
1. _Hamilton DAGs can be called within orchestration system tasks._
-See the [Hamilton + Airflow example](https://blog.dagworks.io/p/supercharge-your-airflow-dag-with). The integration is generally trivial -- all you have to do
+See the [Apache Hamilton + Airflow example](https://blog.dagworks.io/p/supercharge-your-airflow-dag-with). The integration is generally trivial -- all you have to do
is call out to the hamilton library within your task. If your orchestrator supports python, then you're good to go. Some pseudocode (if your orchestrator handles scripts like airflow):
```python
@@ -65,7 +65,7 @@ Examples include:
- [Feast](https://feast.dev/)
- [Tecton](https://tecton.ai/)
-One can think of Hamilton as a being your "feature definition store", where "store" is code + git. While it does
+One can think of Apache Hamilton as a being your "feature definition store", where "store" is code + git. While it does
not provide all the capabilities of a standard feature store, it provides a source of truth for the code that
generated the features, and can be run in a portable method. *So*, if your desire is just to be able to run the same
code in different environments, and have an online/offline store of features, you can use hamilton both to save the
@@ -81,7 +81,7 @@ Also note that joins and aggregations can get tricky. We often recommend using o
definition" i.e. functions decorated with `@config.when`, to either load up the non-online-friendly features
from a feature store or do an external lookup to simulate an online join.
-We expect Hamilton to play a prominent role in the way feature stores work in the future.
+We expect Apache Hamilton to play a prominent role in the way feature stores work in the future.
### Data Science Ecosystems/ML platforms
Examples include:
@@ -93,9 +93,9 @@ Examples include:
- etc.
We've kind of grouped a whole suite of platforms into the same bucket here. These
-tend to have a lot of capabilities all related to ML. Hamilton can be used in conjunction with these
-platforms in a variety of ways. For example, you can use Hamilton to generate features for a model
-that you train in one of these platforms. Or you can use Hamilton to generate a model using the
+tend to have a lot of capabilities all related to ML. Apache Hamilton can be used in conjunction with these
+platforms in a variety of ways. For example, you can use Apache Hamilton to generate features for a model
+that you train in one of these platforms. Or you can use Apache Hamilton to generate a model using the
platform's compute, and then save the model to the platform's registry.
### Registries / Experiment Tracking
@@ -105,16 +105,16 @@ Examples include:
- [DVC](https://dvc.org/)
Most pipelines have a "reverse ETL problem" -- they need to get the results of the pipeline into a some
-sort of datastore or registry. Hamilton can be used in conjunction with these tools as the glue code
-that helps everything work together. For example, you can use Hamilton to generate a model
-and then store metrics computed by Hamilton to one of these "destinations".
+sort of datastore or registry. Apache Hamilton can be used in conjunction with these tools as the glue code
+that helps everything work together. For example, you can use Apache Hamilton to generate a model
+and then store metrics computed by Apache Hamilton to one of these "destinations".
There are three main ways to integrate with these tools:
- - inside a function that Hamilton orchestrates
- - outside Hamilton (e.g. in a script that calls Hamilton)
+ - inside a function that Apache Hamilton orchestrates
+ - outside Apache Hamilton (e.g. in a script that calls Apache Hamilton)
- using "materializers" (see [materializers](../reference/io/index.rst)) (see [this blog](https://blog.dagworks.io/p/separate-data-io-from-transformation)).
-See this [ML reference post](https://blog.dagworks.io/p/from-dev-to-prod-a-ml-pipeline-reference) for examples of how to use Hamilton with these tools.
+See this [ML reference post](https://blog.dagworks.io/p/from-dev-to-prod-a-ml-pipeline-reference) for examples of how to use Apache Hamilton with these tools.
### Python Dataframe/manipulation Libraries
Examples include:
@@ -124,9 +124,9 @@ Examples include:
- [polars](https://www.pola.rs/)
- [duckdb](https://duckdb.org/)
-Hamilton works with any python dataframe/manipulation oriented libraries.
+Apache Hamilton works with any python dataframe/manipulation oriented libraries.
See our [examples folder](https://github.com/apache/hamilton/tree/main/examples)
-to see how to use Hamilton with these libraries.
+to see how to use Apache Hamilton with these libraries.
### Python "big data" systems
@@ -139,8 +139,8 @@ Examples include:
- [pandas-on-spark](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html)
These all provide capabilities to either (a) express and execute computation over datasets in python or (b)
-parallelize it. Often both. Hamilton has a variety of integrations with these systems. The basics is that Hamilton
+parallelize it. Often both. Apache Hamilton has a variety of integrations with these systems. The basics is that Apache Hamilton
can make use of these systems to execute the DAG using the [GraphAdapter](../reference/graph-adapters/index.rst) abstraction and [Lifecycle Hooks](../reference/lifecycle-hooks/index.rst).
See our [examples folder](https://github.com/apache/hamilton/tree/main/examples)
-to see how to use Hamilton with these systems.
+to see how to use Apache Hamilton with these systems.
diff --git a/docs/get-started/your-first-dataflow.rst b/docs/get-started/your-first-dataflow.rst
index 4339824b..9e8799a1 100644
--- a/docs/get-started/your-first-dataflow.rst
+++ b/docs/get-started/your-first-dataflow.rst
@@ -34,8 +34,8 @@ An astute observer might ask the following questions:
#. **Why do the parameter names clash with the function names?** This is core to how hamilton works. It utilizes dependency injection to create a DAG of computation. Parameter names tell the framework where your function gets its data.
#. **OK, if the parameter names determine the source of the data, why have we not defined defined `spend` or `signups` as functions?** This is OK, as we will provide this data as an input when we actually want to materialize our functions. The DAG doesn't have to be complete when it is compiled.
-#. **Why is there no main line to call these functions?** Good observation. In fact, we never will call them (directly)! This is one of the core principles of Hamilton. You write individual transforms and the rest is handled by the framework. More on that next.
-#. **The functions all output pandas series. What if I don't want to use series?** You don't have to! Hamilton is not opinionated on the data type you use. The following are all perfectly valid as well (and we support dask/spark/ray/other distributed frameworks).
+#. **Why is there no main line to call these functions?** Good observation. In fact, we never will call them (directly)! This is one of the core principles of Apache Hamilton. You write individual transforms and the rest is handled by the framework. More on that next.
+#. **The functions all output pandas series. What if I don't want to use series?** You don't have to! Apache Hamilton is not opinionated on the data type you use. The following are all perfectly valid as well (and we support dask/spark/ray/other distributed frameworks).
Let's add a few more functions to our ``my_functions.py`` file:
@@ -126,11 +126,11 @@ And you should see the following output:
2022-02-06 50 400 43.333333 0.125
Not only is your spend to signup ratio decreasing exponentially (your product is going viral!), but you've also
-successfully run your first Hamilton Dataflow. Kudos!
+successfully run your first Apache Hamilton Dataflow. Kudos!
See, wasn't that quick and easy?
Note: if you're ever like "why are things taking a while to execute?", then you might have too much
-in your python environment and Hamilton is auto-loading all the extensions. You can disable this by
+in your python environment and Apache Hamilton is auto-loading all the extensions. You can disable this by
setting the environment variable ``HAMILTON_AUTOLOAD_EXTENSIONS=0`` or programmatically via
``from hamilton import registry; registry.disable_autoload()`` - for more see :doc:`../how-tos/extensions-autoloading`.
diff --git a/docs/hamilton-ui/index.rst b/docs/hamilton-ui/index.rst
index 9f14820f..8ebe9c35 100644
--- a/docs/hamilton-ui/index.rst
+++ b/docs/hamilton-ui/index.rst
@@ -1,5 +1,5 @@
===========
-Hamilton UI
+Apache Hamilton UI
===========
Reference
diff --git a/docs/hamilton-ui/ui.rst b/docs/hamilton-ui/ui.rst
index 1d9179e9..e68b53a6 100644
--- a/docs/hamilton-ui/ui.rst
+++ b/docs/hamilton-ui/ui.rst
@@ -2,22 +2,22 @@
UI Overview
--------------------
-Hamilton comes with a fully open-source UI that can be run both for local deployment and on a remote server.
+Apache Hamilton comes with a fully open-source UI that can be run both for local deployment and on a remote server.
The UI consists of the following features:
1. Telemetry for hamilton executions -- both on the history of executions and the data itself.
2. A feature/artifact catalog for browsing/connecting executions of nodes -> results.
3. A dataflow (i.e. DAG) visualizer for exploring and looking at your code, and determining lineage.
-4. A project explorer for viewing curating projects and viewing versions of your Hamilton dataflows.
+4. A project explorer for viewing curating projects and viewing versions of your Apache Hamilton dataflows.
-In short, the Hamilton UI aims to combine a large swath of MLOps/data observability systems in one simple application.
+In short, the Apache Hamilton UI aims to combine a large swath of MLOps/data observability systems in one simple application.
.. image:: ../_static/hamilton_ui.jpeg
- :alt: Hamilton UI
+ :alt: Apache Hamilton UI
---
-The Hamilton UI has two modes:
+The Apache Hamilton UI has two modes:
1. Run locally using sqlite3
2. Run on docker images with postgres (meant for deployment)
@@ -41,7 +41,7 @@ for full scalability and a multi-read/write db.
Docker/Deployed Mode
--------------------
-The Hamilton UI can be contained within a set of Docker images. You launch with `docker-compose <https://docs.docker.com/compose/>`_, and it will start up the UI, the backend server,
+The Apache Hamilton UI can be contained within a set of Docker images. You launch with `docker-compose <https://docs.docker.com/compose/>`_, and it will start up the UI, the backend server,
and a Postgres database. If you'd like a quick overview of some of the features, you can watch the following:
.. raw:: html
@@ -64,7 +64,7 @@ If you'd like a video walkthrough on getting set up, you can watch the following
As prerequisites, you will need to have Docker installed -- you can follow instructions `here <https://docs.docker.com/engine/install/>`_.
-1. Clone the Hamilton repository locally
+1. Clone the Apache Hamilton repository locally
.. code-block:: bash
@@ -133,9 +133,9 @@ chart contribution!
Running on Snowflake
---------------------
-You can run the Hamilton UI on Snowflake Container Services. For a detailed guide, see the blog post
-`Observability of Python code and application logic with Hamilton UI on Snowflake Container Services <https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635>`_ by
-`Greg Kantyka <ht...@pkantyka>`_ and the `Hamilton Snowflake Example <https://github.com/apache/hamilton/tree/main/examples/snowflake/hamilton_ui>`_.
+You can run the Apache Hamilton UI on Snowflake Container Services. For a detailed guide, see the blog post
+`Observability of Python code and application logic with Apache Hamilton UI on Snowflake Container Services <https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635>`_ by
+`Greg Kantyka <ht...@pkantyka>`_ and the `Apache Hamilton Snowflake Example <https://github.com/apache/hamilton/tree/main/examples/snowflake/hamilton_ui>`_.
-----------
Get started
@@ -156,9 +156,9 @@ Then, navigate to the project page (dashboard/projects), in the running UI, and
Remember the project ID -- you'll use it for the next steps.
-Existing Hamilton Code
+Existing Apache Hamilton Code
----------------------
-Add the following adapter to your code if you have existing Hamilton code:
+Add the following adapter to your code if you have existing Apache Hamilton code:
.. code-block:: python
@@ -182,13 +182,13 @@ Add the following adapter to your code if you have existing Hamilton code:
Then run your DAG, and follow the links in the logs! Note that the link is correct if you're using
the local mode -- if you're on postgres it links to 8241 (but you'll want to follow it to 8241).
-I need some Hamilton code to run
+I need some Apache Hamilton code to run
--------------------------------
-If you don't have Hamilton code to run this with, you can run Hamilton UI example under `examples/hamilton_ui <https://github.com/apache/hamilton/tree/main/examples/hamilton_ui>`_:
+If you don't have Apache Hamilton code to run this with, you can run Apache Hamilton UI example under `examples/hamilton_ui <https://github.com/apache/hamilton/tree/main/examples/hamilton_ui>`_:
.. code-block:: bash
- # we assume you're in the Hamilton repository root
+ # we assume you're in the Apache Hamilton repository root
cd examples/hamilton_ui
# make sure you have the right python packages installed
pip install -r requirements.txt
@@ -283,7 +283,7 @@ HamiltonTracker constructor. By default, these are set to `localhost:8241/8242`.
Changing behavior of what is captured
-------------------------------------
-By default, a lot is captured and sent to the Hamilton UI.
+By default, a lot is captured and sent to the Apache Hamilton UI.
Here are a few options that can change that - these can be found
in `hamilton_sdk.tracking.constants`. You can either change the defaults by
diff --git a/docs/hamilton-vscode/language_server.rst b/docs/hamilton-vscode/language_server.rst
index b84223aa..f04c0560 100644
--- a/docs/hamilton-vscode/language_server.rst
+++ b/docs/hamilton-vscode/language_server.rst
@@ -4,17 +4,17 @@ Language Server
.. warning::
- The Hamilton Language Server is an experimental feature under active development. Edge cases, evolving features, and partial documentation are to be expected. Please open a GitHub issue or reach out on Slack for troubleshooting!
+ The Apache Hamilton Language Server is an experimental feature under active development. Edge cases, evolving features, and partial documentation are to be expected. Please open a GitHub issue or reach out on Slack for troubleshooting!
-The Hamilton Language Server is an implementation of the `Language Server Protocol (LSP) <https://microsoft.github.io/language-server-protocol/>`_. It is designed to power the :doc:`Hamilton VSCode extension <./vscode_extension>` which can be installed directly from the `VSCode marketplace <https://marketplace.visualstudio.com/items?itemName=DAGWorks.hamilton-vsc>`_.
+The Apache Hamilton Language Server is an implementation of the `Language Server Protocol (LSP) <https://microsoft.github.io/language-server-protocol/>`_. It is designed to power the :doc:`Apache Hamilton VSCode extension <./vscode_extension>` which can be installed directly from the `VSCode marketplace <https://marketplace.visualstudio.com/items?itemName=DAGWorks.hamilton-vsc>`_.
Language servers power IDE features like completion suggestion, go to definition, collect document symbols, etc. The LSP standard was established to make servers portable across IDE frontends (e.g., VSCode, PyCharm, Emacs). `Learn more <https://code.visualstudio.com/api/language-extensions/language-server-extension-guide>`_.
Installation
------------
-If you're using the Hamilton VSCode extension, you will prompted to install the language server if it's not found. Simply click the button and it will install it in your current Python interpreter.
+If you're using the Apache Hamilton VSCode extension, you will prompted to install the language server if it's not found. Simply click the button and it will install it in your current Python interpreter.
You can also manually install the language server in your Python environment via
@@ -26,6 +26,6 @@ You can also manually install the language server in your Python environment via
Developers
----------
-If you want to dig in the internals of the language server and integrate it with another IDE, you can find the source code in the ``dev tools/`` section of the `Hamilton GitHub repository <https://github.com/apache/hamilton/tree/main/dev_tools/lsp>`_. It is also directly available on PyPi at `sf-hamilton-lsp <https://pypi.org/project/sf-hamilton-lsp/>`_.
+If you want to dig in the internals of the language server and integrate it with another IDE, you can find the source code in the ``dev tools/`` section of the `Apache Hamilton GitHub repository <https://github.com/apache/hamilton/tree/main/dev_tools/lsp>`_. It is also directly available on PyPi at `sf-hamilton-lsp <https://pypi.org/project/sf-hamilton-lsp/>`_.
Note that the package name is ``hamilton_lsp`` when used directly via Python code.
diff --git a/docs/hamilton-vscode/vscode_extension.rst b/docs/hamilton-vscode/vscode_extension.rst
index bcfe0ef0..b1d6ff25 100644
--- a/docs/hamilton-vscode/vscode_extension.rst
+++ b/docs/hamilton-vscode/vscode_extension.rst
@@ -1,12 +1,12 @@
-------------------------
-Hamilton VSCode
+Apache Hamilton VSCode
-------------------------
.. warning::
- The Hamilton VSCode extension is an experimental feature under active development. Edge cases, evolving features, and partial documentation are to be expected. Please open a GitHub issue or reach out on Slack for troubleshooting!
+ The Apache Hamilton VSCode extension is an experimental feature under active development. Edge cases, evolving features, and partial documentation are to be expected. Please open a GitHub issue or reach out on Slack for troubleshooting!
-The Hamilton VSCode extension enables interactive dataflow development in VSCode. This developer productivity tool helps your editor understand how Hamilton works (code completion, symbol navigation, etc.). It is powered by the :doc:`Hamilton Language Server <./language_server>` and can be installed directly from the `VSCode marketplace <https://marketplace.visualstudio.com/items?itemName=DAGWorks.hamilton-vsc>`_.
+The Apache Hamilton VSCode extension enables interactive dataflow development in VSCode. This developer productivity tool helps your editor understand how Apache Hamilton works (code completion, symbol navigation, etc.). It is powered by the :doc:`Apache Hamilton Language Server <./language_server>` and can be installed directly from the `VSCode marketplace <https://marketplace.visualstudio.com/items?itemName=DAGWorks.hamilton-vsc>`_.
Features
--------
@@ -36,7 +36,7 @@ Get completion suggestions when defining new nodes. It will even insert the appr
Outline
~~~~~~~~~~~~~~~
-The **OUTLINE** menu now displays a **Hamilton VSCode** entry. Nodes and inputs from the current Python file are listed and denoted by distinct icons.
+The **OUTLINE** menu now displays a **Apache Hamilton VSCode** entry. Nodes and inputs from the current Python file are listed and denoted by distinct icons.
.. image:: ./_screenshots/symbol_outline.png
:alt: Browser
@@ -54,7 +54,7 @@ When entering symbol navigation (``CTRL+SHIFT+O``), you can jump directly to any
Extension walkthrough
~~~~~~~~~~~~~~~~~~~~~
-Under the Hamilton menu (the icon at the top), you can find a list of buttons. Selecting ``Walkthrough`` and then ``Get started with Hamilton`` will launch an interactive menu to get you set up along with some tips.
+Under the Apache Hamilton menu (the icon at the top), you can find a list of buttons. Selecting ``Walkthrough`` and then ``Get started with Apache Hamilton`` will launch an interactive menu to get you set up along with some tips.
.. image:: ./_screenshots/links_walkthrough.png
:alt: Browser
@@ -73,5 +73,5 @@ There are many features that we'd be interested in implementing. Let us know on
- Rename: rename a node across locations (can be tricky when mentioned in a decorator)
- Support dataflows spanning multiple modules
- Configure the visualization (i.e., match the Python features)
-- Integrate with the Hamilton UI (e.g., click a node to open it's Hamilton UI page and see execution details)
-- Visualize notebook cells using the Hamilton Jupyter extension (seems possible)
+- Integrate with the Apache Hamilton UI (e.g., click a node to open it's Apache Hamilton UI page and see execution details)
+- Visualize notebook cells using the Apache Hamilton Jupyter extension (seems possible)
diff --git a/docs/how-tos/cli-reference.md b/docs/how-tos/cli-reference.md
index 342e1bbc..1165e895 100644
--- a/docs/how-tos/cli-reference.md
+++ b/docs/how-tos/cli-reference.md
@@ -1,9 +1,9 @@
# Command line interface
-This page covers the Hamilton CLI. It is built directly from the CLI, but note that the command `hamilton --help` always provide the most accurate documentation.
+This page covers the Apache Hamilton CLI. It is built directly from the CLI, but note that the command `hamilton --help` always provide the most accurate documentation.
## Installation
-The dependencies for the Hamilton CLI can be installed via
+The dependencies for the Apache Hamilton CLI can be installed via
```console
pip install sf-hamilton[cli]
diff --git a/docs/how-tos/extensions-autoloading.rst b/docs/how-tos/extensions-autoloading.rst
index 01e8782d..84005be5 100644
--- a/docs/how-tos/extensions-autoloading.rst
+++ b/docs/how-tos/extensions-autoloading.rst
@@ -2,21 +2,21 @@
Extension autoloading
=====================
-Under ``hamilton.plugins``, there are many modules named ``*_extensions`` (e.g., ``hamilton.plugins.pandas_extensions``, ``hamilton.plugins.mlflow_extensions``). They implement Hamilton features for 3rd party libraries, including ``@extract_columns``, materializers (``to.parquet``, ``from_.mlflow``), and more.
+Under ``hamilton.plugins``, there are many modules named ``*_extensions`` (e.g., ``hamilton.plugins.pandas_extensions``, ``hamilton.plugins.mlflow_extensions``). They implement Apache Hamilton features for 3rd party libraries, including ``@extract_columns``, materializers (``to.parquet``, ``from_.mlflow``), and more.
Autoloading behavior
--------------------
-By default, Hamilton attempts to load all extensions one-by-one. This means that as you have more Python packages in your environment (e.g., ``pandas``, ``pyspark``, ``mlflow``, ``xgboost``), importing Hamilton appears to become slower because it actually imports many packages.
+By default, Apache Hamilton attempts to load all extensions one-by-one. This means that as you have more Python packages in your environment (e.g., ``pandas``, ``pyspark``, ``mlflow``, ``xgboost``), importing Apache Hamilton appears to become slower because it actually imports many packages.
-This behavior can be less desirable when your Hamilton dataflow doesn't use any of these packages, but you need them in your Python environment nonetheless. For example, if only ``pandas`` is needed for your dataflow, but you have ``mlflow`` and ``xgboost`` in your environment their respective extensions will be loaded each time.
+This behavior can be less desirable when your Apache Hamilton dataflow doesn't use any of these packages, but you need them in your Python environment nonetheless. For example, if only ``pandas`` is needed for your dataflow, but you have ``mlflow`` and ``xgboost`` in your environment their respective extensions will be loaded each time.
Disable autoloading
--------------------
-Disabling extension autoloading allows to import Hamilton without any extensions, which can reduce import time from 2-3 sec to less than 0.5 sec. This speedup is welcomed when you need to restart a notebook's kernel often or you're operating in a low RAM environment (some Python packages are larger than 50Mbs).
+Disabling extension autoloading allows to import Apache Hamilton without any extensions, which can reduce import time from 2-3 sec to less than 0.5 sec. This speedup is welcomed when you need to restart a notebook's kernel often or you're operating in a low RAM environment (some Python packages are larger than 50Mbs).
There are three ways to opt-out: programmatically, environment variables, configuration file. You must opt-out before having any other ``hamilton`` import.
@@ -53,7 +53,7 @@ Programmatically in Jupyter notebooks
3. Configuration file
~~~~~~~~~~~~~~~~~~~~~
-Using the following command disables autoloading via the configuration file ``./hamilton.conf``. Hamilton won't autoload extensions anymore (i.e., you won't need to use approach 1 or 2 each time).
+Using the following command disables autoloading via the configuration file ``./hamilton.conf``. Apache Hamilton won't autoload extensions anymore (i.e., you won't need to use approach 1 or 2 each time).
.. code-block:: console
diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst
index 06f2fbc9..2f74e243 100644
--- a/docs/how-tos/index.rst
+++ b/docs/how-tos/index.rst
@@ -2,7 +2,7 @@
User Guide
==============
-This portion of the documentation goes over the set of common examples for Hamilton usage, so you can apply
+This portion of the documentation goes over the set of common examples for Apache Hamilton usage, so you can apply
it to your day-to-day work. Each one corresponds to an example in the `examples <https://github.com/apache/hamilton/tree/main/examples>`_
directory. If there's an example you want but don't see, reach out or open an issue on github -- we're always looking to add more.
diff --git a/docs/how-tos/llm-workflows.md b/docs/how-tos/llm-workflows.md
index 6e739ae7..d3fca665 100644
--- a/docs/how-tos/llm-workflows.md
+++ b/docs/how-tos/llm-workflows.md
@@ -1,10 +1,10 @@
# LLM workflows
-Hamilton is great for describing dataflows, and a lot of "actions" you want
+Apache Hamilton is great for describing dataflows, and a lot of "actions" you want
an "agent" to perform can be described as one, e.g. create an embedding
of some passed in text, query a vector database, find the nearest documents, etc.
-The benefit of using Hamilton within an LLM Powered app is that:
+The benefit of using Apache Hamilton within an LLM Powered app is that:
1. you can visualize the dataflow.
2. you can easily test, modify, compose, and reuse dataflows. For example,
you can easily test the dataflow that creates an embedding of some text
@@ -12,12 +12,12 @@ The benefit of using Hamilton within an LLM Powered app is that:
3. you can easily swap out the implementation details of components surgically. For example,
you can swap out the vector database client based on configuration, this helps in ensuring
you can quickly and easily modify/update your dataflow and have confidence around the impact of that change.
-4. you can use functionality like runtime data quality checks/extend Hamilton's capabilities with your own needs to inject/augment
+4. you can use functionality like runtime data quality checks/extend Apache Hamilton's capabilities with your own needs to inject/augment
your dataflow with additional functionality, e.g. caching, logging, etc.
5. you can request the intermediate outputs of a dataflow by requesting it as output without any surgery required to
any of your code to do so. This is useful for debugging.
-The following examples show how to use Hamilton for LLM workflows:
+The following examples show how to use Apache Hamilton for LLM workflows:
* [How to use "OpenAI functions" with a Knowledge Base](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/knowledge_retrieval/)
* [Modular LLM Stack](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/modular_llm_stack) with [blog post](https://blog.dagworks.io/p/building-a-maintainable-and-modular)
diff --git a/docs/how-tos/load-data.rst b/docs/how-tos/load-data.rst
index 85405c4c..d0c9675a 100644
--- a/docs/how-tos/load-data.rst
+++ b/docs/how-tos/load-data.rst
@@ -2,8 +2,8 @@
Loading data
==================
-While we've been injecting data in from the driver in previous examples, Hamilton functions are fully capable of loading their own data.
-In the following example, we'll show how to use Hamilton to:
+While we've been injecting data in from the driver in previous examples, Apache Hamilton functions are fully capable of loading their own data.
+In the following example, we'll show how to use Apache Hamilton to:
1. Load data from an external source (CSV file and duckdb database)
2. Alter the source of data depending on how the DAG is parameterized/created
diff --git a/docs/how-tos/microservice.rst b/docs/how-tos/microservice.rst
index 370f7dfb..6d0aad40 100644
--- a/docs/how-tos/microservice.rst
+++ b/docs/how-tos/microservice.rst
@@ -2,7 +2,7 @@
Microservice
==============================
-While we've mainly been discussing running Hamilton in a batch environment, it can easily be used
+While we've mainly been discussing running Apache Hamilton in a batch environment, it can easily be used
in a microservice/online setting. This is valuable if you want insight into exactly how your endpoints
transform/load data, or if you want to execute the same transforms you did in batch in an online setting.
diff --git a/docs/how-tos/ml-training.rst b/docs/how-tos/ml-training.rst
index 36c7e75f..68d8c5b4 100644
--- a/docs/how-tos/ml-training.rst
+++ b/docs/how-tos/ml-training.rst
@@ -2,11 +2,11 @@
Model training
===============================
-As Hamilton is a generic library for representing dataflows in pandas, it can be used for a wide array of tasks.
+As Apache Hamilton is a generic library for representing dataflows in pandas, it can be used for a wide array of tasks.
One of the more common applications is using hamilton for training, testing, and executing machine learning models,
all the way from feature-engineering through training and inference.
-The following two examples show how to use Hamilton to model an entire ML pipeline:
+The following two examples show how to use Apache Hamilton to model an entire ML pipeline:
1. A `classification pipeline <https://github.com/apache/hamilton/tree/main/examples/model_examples/scikit-learn>`_ for the iris dataset using scikit-learn
2. An `implementation <https://github.com/apache/hamilton/tree/main/examples/model_examples/time-series>`_ of the m5 kaggle competition to do time-series forecasting on unit sales for using Walmart data.
diff --git a/docs/how-tos/pre-commit-hooks.md b/docs/how-tos/pre-commit-hooks.md
index 52b868fa..b7c519c0 100644
--- a/docs/how-tos/pre-commit-hooks.md
+++ b/docs/how-tos/pre-commit-hooks.md
@@ -1,7 +1,7 @@
# pre-commit hooks
-## Use pre-commit hooks for safer Hamilton code changes
+## Use pre-commit hooks for safer Apache Hamilton code changes
-This page gives an introduction to pre-commit hooks and how to use custom hooks to validate your Hamilton code.
+This page gives an introduction to pre-commit hooks and how to use custom hooks to validate your Apache Hamilton code.
## What are pre-commit hooks?
A pre-commit hook is a script or command that's executed automatically before making a commit. The goal of these hooks is to standardize code formatting and catch erroneous code before being committed. For example, popular hooks include ensuring files have no syntax errors, sorting imports, and normalizing line breaks.
@@ -50,8 +50,8 @@ The preferred way of working with pre-commit hooks is through the [pre-commit](h
pre-commit run --all-files
```
-## Custom Hamilton pre-commit hooks
-pre-commit hooks are great developer tools, but off-the-shelf solutions aren't aware of the Hamilton framework. Hence, we developed a pre-commit hook to help you author Hamilton dataflows! Under the hood, they leverage the `hamilton` CLI, so if you are unfamiliar with it, feel free to install it and view the `--help` messages.
+## Custom Apache Hamilton pre-commit hooks
+pre-commit hooks are great developer tools, but off-the-shelf solutions aren't aware of the Apache Hamilton framework. Hence, we developed a pre-commit hook to help you author Apache Hamilton dataflows! Under the hood, they leverage the `hamilton` CLI, so if you are unfamiliar with it, feel free to install it and view the `--help` messages.
```console
pip install sf-hamilton[cli]
@@ -59,13 +59,13 @@ hamilton --help
```
### Checking dataflow definition
-Hamilton doesn't have many syntactic constraints, but there's a few things we want to catch:
+Apache Hamilton doesn't have many syntactic constraints, but there's a few things we want to catch:
- functions parameters and return are type annotated
- a node consistently has the same type (e.g., a parameter in multiple functions)
- functions with a name starting with underscore (`_`) are ignored from the dataflow
- functions with a `@config` decorator received a trailing double underscore with a suffix (e.g., `hello__weekday()`, `hello__weekend()`)
-Instead of reimplementing this logic, we can try to build the Hamilton Driver with the command `hamilton build MODULES` and catch errors. This also ensures the verification is always in sync with the actual build mechanism. This hook will help prevent us from committing invalid dataflow definitions.
+Instead of reimplementing this logic, we can try to build the Apache Hamilton Driver with the command `hamilton build MODULES` and catch errors. This also ensures the verification is always in sync with the actual build mechanism. This hook will help prevent us from committing invalid dataflow definitions.
### Checking dataflow paths
A dataflow definition might be valid, but it might break paths in unexpected ways. The command `hamilton validate` (which internally uses `Driver.validate_execution()`) can check if a node is reachable.
@@ -110,7 +110,7 @@ In `v1`, the dataflow could be validated for `C` without any inputs. Now, a deve
pre-commit hooks can prevent commits from breaking a core path, but you should use unit and integration tests for more robust checks.
```
-## Add Hamilton pre-commit to your project
+## Add Apache Hamilton pre-commit to your project
We alluded to the relationship between pre-commit hooks and the `hamilton` command line tool. In fact, the basic hook is designed to take a list of `hamilton` commands and will execute them in order when hooks are triggered.
To use them, add this snippet to your `.pre-commit-config.yaml` and adapt it to your project:
@@ -120,7 +120,7 @@ To use them, add this snippet to your `.pre-commit-config.yaml` and adapt it to
rev: v0.1.2 # use a ref >= 0.1.2
hooks:
- id: cli-command
- name: Hamilton CLI command
+ name: Apache Hamilton CLI command
args: [ # list of CLI commands to execute
hamilton build my_module.py,
hamilton build my_module2.py,
diff --git a/docs/how-tos/run-data-quality-checks.rst b/docs/how-tos/run-data-quality-checks.rst
index f2094d1b..c85f56b7 100644
--- a/docs/how-tos/run-data-quality-checks.rst
+++ b/docs/how-tos/run-data-quality-checks.rst
@@ -2,7 +2,7 @@
Data quality
=======================
-Hamilton comes with data quality included out of the box.
+Apache Hamilton comes with data quality included out of the box.
While you can read more about this in the :doc:`API reference <../reference/decorators/index/>`, we have a few examples to help get you started.
The following two examples showcase a similar workflow, one using the vanilla hamilton data quality decorator, and the other using the pandera integration.
diff --git a/docs/how-tos/scale-up.rst b/docs/how-tos/scale-up.rst
index 28e9ebfb..4a4df4db 100644
--- a/docs/how-tos/scale-up.rst
+++ b/docs/how-tos/scale-up.rst
@@ -2,9 +2,9 @@
Scaling computation
=====================
-Hamilton enables a variety of tools for allowing you to scale your data processing by integrating with third-party libraries.
+Apache Hamilton enables a variety of tools for allowing you to scale your data processing by integrating with third-party libraries.
-Specifically, we have four examples that show how to scale Hamilton both by parallelizing transformations (ray and dask) and running
+Specifically, we have four examples that show how to scale Apache Hamilton both by parallelizing transformations (ray and dask) and running
on larger, distributed datasets (pandas on spark, pyspark map UDFs).
1. Integrating hamilton with `pandas on spark <https://github.com/apache/hamilton/tree/main/examples/spark/pandas_on_spark>`_.
diff --git a/docs/how-tos/use-for-feature-engineering.rst b/docs/how-tos/use-for-feature-engineering.rst
index 2724466e..da249c90 100644
--- a/docs/how-tos/use-for-feature-engineering.rst
+++ b/docs/how-tos/use-for-feature-engineering.rst
@@ -2,12 +2,12 @@
Feature engineering
==========================================
-Hamilton's roots are in time-series offline feature engineering. But it can be used for any type of feature engineering:
-offline, streaming, online. All our examples are oriented towards Pandas, but rest assured, you can use Hamilton with
+Apache Hamilton's roots are in time-series offline feature engineering. But it can be used for any type of feature engineering:
+offline, streaming, online. All our examples are oriented towards Pandas, but rest assured, you can use Apache Hamilton with
any python objects, e.g. numpy, polars, and even pyspark.
-Here's a 20 minute video (`slides <https://github.com/skrawcz/talks/files/9759661/FS.Summit.2022.-.Hamilton.pdf>`__), with
-brief backstory on Hamilton, and an overview (at around the 8:52 mark) of how to use it for feature engineering which
+Here's a 20 minute video (`slides <https://github.com/skrawcz/talks/files/9759661/FS.Summit.2022.-.Apache Hamilton.pdf>`__), with
+brief backstory on Apache Hamilton, and an overview (at around the 8:52 mark) of how to use it for feature engineering which
was presented at the Feature Store Summit 2022:
.. raw:: html
@@ -16,17 +16,17 @@ was presented at the Feature Store Summit 2022:
Otherwise here we present a high level overview and then direct users to the examples folder for more details. We suggest
reading the Offline Feature Engineering section first, since it's the most common use case, and helps explain the
-python module structure you should be going for with Hamilton. If you need more guidance here, please reach out to us on
+python module structure you should be going for with Apache Hamilton. If you need more guidance here, please reach out to us on
`slack <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`__.
Offline Feature Engineering
---------------------------
-To use Hamilton for offline feature engineering, a common pattern is:
+To use Apache Hamilton for offline feature engineering, a common pattern is:
1. create a data_loader module(s) that loads the data from the source(s) (e.g. a database, a csv file, etc.).
2. create feature transform module(s) that transform the data into features.
3. create a data set module(s) that combines the data_loader and feature transform modules if you want to connect fitting \
- a model with Hamilton. Or, you do this data set definition in your driver code.
+ a model with Apache Hamilton. Or, you do this data set definition in your driver code.
Here is a sketch of the above pattern:
@@ -51,32 +51,32 @@ Here is a sketch of the above pattern:
...
-Hamilton Example
+Apache Hamilton Example
^^^^^^^^^^^^^^^^
We do not provide a specific example here, since most of the examples in the examples folder fall under this category.
Some examples to browse:
* `Hello World <https://github.com/apache/hamilton/tree/main/examples/hello_world>`__ shows the basics of how to
- use Hamilton.
+ use Apache Hamilton.
* `Data Quality <https://github.com/apache/hamilton/tree/main/examples/data_quality>`__ shows how to incorporate
runtime data quality checks into your feature engineering pipeline.
* `Time-series Kaggle Example <https://github.com/apache/hamilton/tree/main/examples/model_examples/time-series>`__
shows one way to structure your code to ingest, create features, and fit a model.
* `Feature engineering in multiple contexts <https://github.com/apache/hamilton/tree/main/examples/feature_engineering/feature_engineering_multiple_contexts>`__
- helps show how you can use Hamilton in multiple contexts reusing code where possible, e.g. offline, & online.
+ helps show how you can use Apache Hamilton in multiple contexts reusing code where possible, e.g. offline, & online.
* `PySpark UDF Map Examples <https://github.com/apache/hamilton/tree/main/examples/spark/pyspark_udfs>`__
- shows how to use Hamilton to encode map operations for use with PySpark.
+ shows how to use Apache Hamilton to encode map operations for use with PySpark.
Streaming Feature Engineering
-----------------------------
-Right now, there is no specific streaming support. Instead, we model the problem as we would for offline. Hamilton
-has an `inputs=` argument to the `execute()` function in the driver. This allows you to then instantiate a Hamilton
+Right now, there is no specific streaming support. Instead, we model the problem as we would for offline. Apache Hamilton
+has an `inputs=` argument to the `execute()` function in the driver. This allows you to then instantiate a Apache Hamilton
Driver once, and then call `execute()` multiple times with different inputs. Otherwise you'd have a similar python
module structure as for offline feature engineering -- perhaps just dropping the data_loader module since you would
provide the inputs directly to the `execute()` function.
-Here's a sketch of how you might use Hamilton in conjunction with a Kafka Client:
+Here's a sketch of how you might use Apache Hamilton in conjunction with a Kafka Client:
.. code-block:: python
@@ -89,13 +89,13 @@ Here's a sketch of how you might use Hamilton in conjunction with a Kafka Client
# do something / emit back to kafka, etc.
-**Caveats to think about**. Here are some things to think about when using Hamilton for streaming feature engineering:
+**Caveats to think about**. Here are some things to think about when using Apache Hamilton for streaming feature engineering:
- aggregation features, you likely want to understand whether you want to aggregate over the entire stream or just \
the current batch, or load values that were computed offline.
-Hamilton Example
+Apache Hamilton Example
^^^^^^^^^^^^^^^^
Currently we don't have a streaming example. But we are working on it. We direct users to look at the online example
for now, since conceptually from a modularity stand point, things would be set up in a similar way.
@@ -103,26 +103,26 @@ for now, since conceptually from a modularity stand point, things would be set u
Online Feature Engineering
--------------------------
Online feature engineering can be quite simple or quite complex, depending on your situation. However, good news is,
-that Hamilton should be able to help you in any situation. The modularity of Hamilton allows you to swap out implementations
+that Apache Hamilton should be able to help you in any situation. The modularity of Apache Hamilton allows you to swap out implementations
of features easily, as well as override values, and even ask the Driver what features are required from the source data
-to create the features that you want. We think Hamilton can help you keep things simple, but then extend to helping you
+to create the features that you want. We think Apache Hamilton can help you keep things simple, but then extend to helping you
handle more complex situations.
-The basic structure of your python modules, does not change. Depending on whether you want Hamilton to load data from a feature store,
+The basic structure of your python modules, does not change. Depending on whether you want Apache Hamilton to load data from a feature store,
or you have all the data passed in, you just need to appropriately segment your feature transforms into modules, or use
the `@config.*` decorator, to help you segment your feature computation dataflow to give you the flexibility you need.
-*Caveats to think about*. Here are some things to think about when using Hamilton for online feature engineering:
+*Caveats to think about*. Here are some things to think about when using Apache Hamilton for online feature engineering:
- aggregation features, most likely you'll want to load aggregated feature values that were computed offline, rather \
than compute them live.
We skip showing a sketch of structure here, and invite you to look at the examples below.
-Hamilton Example
+Apache Hamilton Example
^^^^^^^^^^^^^^^^
We direct users to look at `Feature engineering in multiple contexts <https://github.com/apache/hamilton/tree/main/examples/feature_engineering/feature_engineering_multiple_contexts>`__
-that currently describes two scenarios around how you could incorporate Hamilton into an online web-service, and have
+that currently describes two scenarios around how you could incorporate Apache Hamilton into an online web-service, and have
it aligned with your batch offline processes. Note, these examples should give you the high level first principles
view of how to do things. Since having something running in production , we didn't want to get too specific.
@@ -133,12 +133,12 @@ The companion example code can be found `here <https://github.com/apache/hamilto
Best Egg Platform Blog Post:
----------------------------
-For an overview of how Best Egg built their feature platform on Hamilton see `this blog <https://blog.dagworks.io/p/building-a-better-feature-platform?r=2cg5z1&utm_campaign=post&utm_medium=web>`__.
+For an overview of how Best Egg built their feature platform on Apache Hamilton see `this blog <https://blog.dagworks.io/p/building-a-better-feature-platform?r=2cg5z1&utm_campaign=post&utm_medium=web>`__.
FAQ
----
-Q. Can I use Hamilton for feature engineering with Feast?
+Q. Can I use Apache Hamilton for feature engineering with Feast?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Yes, you can use Hamilton with Feast. See our [Feast example](https://github.com/apache/hamilton/tree/main/examples/feast) and accompanying [blog post](https://blog.dagworks.io/p/featurization-integrating-hamilton). Typically people use Hamilton on the offline side to compute features that then
+Yes, you can use Apache Hamilton with Feast. See our [Feast example](https://github.com/apache/hamilton/tree/main/examples/feast) and accompanying [blog post](https://blog.dagworks.io/p/featurization-integrating-hamilton). Typically people use Apache Hamilton on the offline side to compute features that then
get pushed to Feast. For the online side it varies as to how to integrate the two.
diff --git a/docs/how-tos/use-in-jupyter-notebook.md b/docs/how-tos/use-in-jupyter-notebook.md
index cb475142..cd44a0da 100644
--- a/docs/how-tos/use-in-jupyter-notebook.md
+++ b/docs/how-tos/use-in-jupyter-notebook.md
@@ -1,15 +1,15 @@
# Jupyter notebooks
-There are two main ways to use Hamilton in a notebook.
+There are two main ways to use Apache Hamilton in a notebook.
1. Dynamically create modules within the notebook.
2. Import modules into the notebook.
## 1 - Dynamically create modules within your notebook
-There's two main ways, using the Hamilton Jupyter magic, or using `ad_hoc_utils` to create a temporary module.
+There's two main ways, using the Apache Hamilton Jupyter magic, or using `ad_hoc_utils` to create a temporary module.
-### Use Hamilton Jupyter Magic
-The Hamilton Jupyter magic allows you to dynamically create a module from a cell in your notebook. This is useful for quick iteration and development.
+### Use Apache Hamilton Jupyter Magic
+The Apache Hamilton Jupyter magic allows you to dynamically create a module from a cell in your notebook. This is useful for quick iteration and development.
Once you're then happy, it's easy to then write out a module with the functions you've developed using `%%writefile` magic.
To load the magic:
@@ -54,7 +54,7 @@ Once you're happy with the functions you've developed, you can then write them o
#### Importing specific functions into cell modules
-If you import parts of modules in a Hamilton Jupyter Magic cell, these will need to be reloaded when changes are made to their source. This can be done either by restarting the kernel or with the help of importlib.reload:
+If you import parts of modules in a Apache Hamilton Jupyter Magic cell, these will need to be reloaded when changes are made to their source. This can be done either by restarting the kernel or with the help of importlib.reload:
```python
%%cell_to_module MODULE_NAME
@@ -95,7 +95,7 @@ temp_module = ad_hoc_utils.create_temporary_module(
log_avg_3wk_spend, module_name='function_example')
```
-You can now treat `temp_module` like a python module and pass it to your driver and use Hamilton like normal:
+You can now treat `temp_module` like a python module and pass it to your driver and use Apache Hamilton like normal:
```python
# Step 3 - add the module to the driver and continue as usual
@@ -111,7 +111,7 @@ Using a "temporary python module" will not enable scaling of computation by usin
## 2 - Importing modules into your notebook
This tutorial can also be found [published on TDS](https://towardsdatascience.com/how-to-iterate-with-hamilton-in-a-notebook-8ec0f85851ed).
-### Step 1 β Install Jupyter & Hamilton <a href="#cb52" id="cb52"></a>
+### Step 1 β Install Jupyter & Apache Hamilton <a href="#cb52" id="cb52"></a>
I assume you already have this step set up. But just in case you donβt:
@@ -125,17 +125,17 @@ Then to start the notebook server it should just be:
### Step 2β Set up the files <a href="#57fe" id="57fe"></a>
1. Start up your Jupyter notebook.
-2. Go to the directory where you want your notebook and Hamilton function module(s) to live.
+2. Go to the directory where you want your notebook and Apache Hamilton function module(s) to live.
3. Create a python file(s). Do that by going to βNew > text fileβ. Itβll open a βfileβ editor view. Name the file and give it a `.py` extension. Once you save it, youβll see that jupyter now provides python syntax highlighting. Keep this tab open, so you can flip back to it to edit this file.
4. Start up a notebook that you will use in another browser tab.
### Step 3β The basic process of iteration <a href="#e434" id="e434"></a>
-At a high level, you will be switching back and forth between your tabs. You will add functions to your Hamilton function python module, and then import/reimport that module into your notebook to get the changes. From there you will then use Hamilton as usual to run and execute things and the notebook for all the standard things you use notebooks for.
+At a high level, you will be switching back and forth between your tabs. You will add functions to your Apache Hamilton function python module, and then import/reimport that module into your notebook to get the changes. From there you will then use Apache Hamilton as usual to run and execute things and the notebook for all the standard things you use notebooks for.
Letβs walk through an example.
-Hereβs a function I added to our Hamilton function module. I named the module `some_functions.py` (obviously choose a better name for your situation).
+Hereβs a function I added to our Apache Hamilton function module. I named the module `some_functions.py` (obviously choose a better name for your situation).
```python
import pandas as pd
@@ -146,7 +146,7 @@ def avg_3wk_spend(spend: pd.Series) -> pd.Series:
return spend.rolling(3).mean()
```
-And hereβs what I set up in my notebook to be able to use Hamilton and import this module:
+And hereβs what I set up in my notebook to be able to use Apache Hamilton and import this module:
Cell 1: This just imports the base things we need; see the pro-tip at the bottom of this page for how to automatically reload changes.
@@ -156,7 +156,7 @@ import pandas as pd
from hamilton import driver
```
-Cell 2: Import your Hamilton function module(s)
+Cell 2: Import your Apache Hamilton function module(s)
```python
# import your hamilton function module(s) here
@@ -172,7 +172,7 @@ importlib.reload(some_functions)
What this will do is reload the module, and therefore make sure the code is up to date for you to use.
-Cell 4: Use Hamilton
+Cell 4: Use Apache Hamilton
```python
config = {}
@@ -183,7 +183,7 @@ df = dr.execute(['avg_3wk_spend'], inputs=input_data)
You should see `foo` printed as an output after running this cell.
-Okay, so letβs now say weβre iterating on our Hamilton functions. Go to your Hamilton function module (`some_functions.py` in this example) in your other browser tab, and change the `print("foo")` to something else, e.g. `print("foo-bar").` Save the file β it should look something like this:
+Okay, so letβs now say weβre iterating on our Apache Hamilton functions. Go to your Apache Hamilton function module (`some_functions.py` in this example) in your other browser tab, and change the `print("foo")` to something else, e.g. `print("foo-bar").` Save the file β it should look something like this:
```python
def avg_3wk_spend(spend: pd.Series) -> pd.Series:
@@ -194,7 +194,7 @@ def avg_3wk_spend(spend: pd.Series) -> pd.Series:
Go back to your notebook, and re-run Cell 3 & Cell 4. You should now see a different output printed, e.g. `foo-bar` .
-Congratulations! You just managed to iterate on Hamilton using a Jupyter notebook!
+Congratulations! You just managed to iterate on Apache Hamilton using a Jupyter notebook!
**To summarize** this is how things ended up looking on my end:
@@ -231,15 +231,15 @@ hamilton_driver.execute(['desired_output1', 'desired_output2'])
You'd then follow the following process:
1. Write your data transformation in the open python module
-2. In the notebook, instantiate a Hamilton driver and test the DAG with a small subset of data. 
-3. Because of %autoreload, the module is reimported with the latest changes each time the Hamilton DAG is executed. This approach prevents out-of-order notebook executions, and functions always reside in clean .py files.
+2. In the notebook, instantiate a Apache Hamilton driver and test the DAG with a small subset of data. 
+3. Because of %autoreload, the module is reimported with the latest changes each time the Apache Hamilton DAG is executed. This approach prevents out-of-order notebook executions, and functions always reside in clean .py files.
Credit: [Thierry Jean's blog post](https://medium.com/@thijean/the-perks-of-creating-dataflows-with-hamilton-36e8c56dd2a).
## Pro-tip: You can import functions directly <a href="#2e10" id="2e10"></a>
-The nice thing about forcing Hamilton functions into a module, is that itβs very easy to re-use in another context. E.g. another notebook, or directly.
+The nice thing about forcing Apache Hamilton functions into a module, is that itβs very easy to re-use in another context. E.g. another notebook, or directly.
For example, it is easy to directly use the functions in the notebook, like so:
diff --git a/docs/how-tos/wrapping-driver.rst b/docs/how-tos/wrapping-driver.rst
index f63d2459..2f8bbb32 100644
--- a/docs/how-tos/wrapping-driver.rst
+++ b/docs/how-tos/wrapping-driver.rst
@@ -1,11 +1,11 @@
Wrapping the Driver
------------------------------
-The APIs that the Hamilton Driver is built on, are considered internal. So it is possible for you to define your own
-driver in place of the stock Hamilton driver, we suggest the following path if you don't like how the current Hamilton
+The APIs that the Apache Hamilton Driver is built on, are considered internal. So it is possible for you to define your own
+driver in place of the stock Apache Hamilton driver, we suggest the following path if you don't like how the current Apache Hamilton
Driver interface is designed:
-`Write a "Wrapper" class that delegates to the Hamilton Driver.`
+`Write a "Wrapper" class that delegates to the Apache Hamilton Driver.`
i.e.
@@ -21,11 +21,11 @@ i.e.
# ...
def my_execute_function(self, arg1, arg2, ...):
- """What actually calls the Hamilton"""
+ """What actually calls the Apache Hamilton"""
dr = driver.Driver(self.constructor_arg, ...)
df = dr.execute(self.outputs)
return self.augmetn(df)
-That way, you can create the right API constructs to invoke Hamilton in your context, and then delegate to the stock
-Hamilton Driver. By doing so, it will ensure that your code continues to work, since we intend to honor the Hamilton
+That way, you can create the right API constructs to invoke Apache Hamilton in your context, and then delegate to the stock
+Apache Hamilton Driver. By doing so, it will ensure that your code continues to work, since we intend to honor the Apache Hamilton
Driver APIs with backwards compatibility as much as possible.
diff --git a/docs/integrations/dbt.rst b/docs/integrations/dbt.rst
index aca3d070..3b611720 100644
--- a/docs/integrations/dbt.rst
+++ b/docs/integrations/dbt.rst
@@ -2,12 +2,12 @@
dbt
=====================
-If you're familiar with DBT, you likely noticed that it can fill a similar role to Hamilton. What DBT does for SQL
-files (organizing functions, providing lineage capabilities, making testing easier), Hamilton does for python functions.
+If you're familiar with DBT, you likely noticed that it can fill a similar role to Apache Hamilton. What DBT does for SQL
+files (organizing functions, providing lineage capabilities, making testing easier), Apache Hamilton does for python functions.
-Many projects span the gap between SQL and python, and Hamilton is a natural next step for an ML workflow after extracting data from DBT.
+Many projects span the gap between SQL and python, and Apache Hamilton is a natural next step for an ML workflow after extracting data from DBT.
-This example shows how you can use DBT's `new python capabilities <https://docs.getdbt.com/docs/build/python-models>`_ to integrate a Hamilton dataflow
+This example shows how you can use DBT's `new python capabilities <https://docs.getdbt.com/docs/build/python-models>`_ to integrate a Apache Hamilton dataflow
with a DBT pipeline.
Find the full, working dbt project `here <https://github.com/apache/hamilton/tree/main/examples/dbt>`_.
diff --git a/docs/integrations/dlt/index.md b/docs/integrations/dlt/index.md
index 2d054384..d21fba0a 100644
--- a/docs/integrations/dlt/index.md
+++ b/docs/integrations/dlt/index.md
@@ -2,25 +2,25 @@
[dlt](https://dlthub.com/) stands for "data load tool". It's an open-source Python library providing a ton of data `Sources` (Slack, Stripe, Google Analytics, Zendesk, etc.) and `Destinations` (S3, Snowflake, BigQuery, Postgres, etc.). `Pipelines` make it easy to connect `Sources` and `Destinations` and provide advanced engineering features such as table normalization, incremental loading, and automatic schema evolution.
-dlt is an "extract and load" tool and Hamilton is a "transform" tool, allowing various usage patterns.
+dlt is an "extract and load" tool and Apache Hamilton is a "transform" tool, allowing various usage patterns.
On this page, you'll learn:
- Extract, Transform, Load (ETL)
- Extract, Load, Transform (ELT)
-- dlt materializer plugin for Hamilton
+- dlt materializer plugin for Apache Hamilton
``` {note}
-See this [blog post](https://blog.dagworks.io/p/slack-summary-pipeline-with-dlt-ibis) for a more detailed discussion about ETL with dlt + Hamilton
+See this [blog post](https://blog.dagworks.io/p/slack-summary-pipeline-with-dlt-ibis) for a more detailed discussion about ETL with dlt + Apache Hamilton
```
## Extract, Transform, Load (ETL)
The key consideration for ETL is that the data has to move twice:
-> ingest raw data (dlt) -> transform (Hamilton) -> store transformed data (dlt)
+> ingest raw data (dlt) -> transform (Apache Hamilton) -> store transformed data (dlt)
1. **Extract**: dlt moves the raw data to a processing server
-2. **Transform**: on the server, Hamilton executes transformations
+2. **Transform**: on the server, Apache Hamilton executes transformations
3. **Load**: dlt moves the final data to its destination (database, dashboard, etc.)
**Pros**
@@ -58,7 +58,7 @@ The key consideration for ETL is that the data has to move twice:
### Transform
-2. Define the Hamilton dataflow of transformations
+2. Define the Apache Hamilton dataflow of transformations
```python
# transform.py
@@ -94,7 +94,7 @@ The key consideration for ETL is that the data has to move twice:
-3. Add the Hamilton dataflow execution code to `run.py`
+3. Add the Apache Hamilton dataflow execution code to `run.py`
```python
# run.py
@@ -119,13 +119,13 @@ The key consideration for ETL is that the data has to move twice:
destination='bigquery',
dataset_name="slack_community_backup"
)
- # pass the results from Hamilton to dlt
+ # pass the results from Apache Hamilton to dlt
data = results["threads"].to_dict(orient="records")
final_load_info = load_pipeline.run(data, table_name="threads")
```
### ETL Summary
-You need to set up your dlt pipeline for raw and transformed data, and define your Hamilton transformation dataflow. Then, your execution code consist of executing the ETL step in sequence. It should look like this:
+You need to set up your dlt pipeline for raw and transformed data, and define your Apache Hamilton transformation dataflow. Then, your execution code consist of executing the ETL step in sequence. It should look like this:
```python
# run.py
@@ -162,12 +162,12 @@ final_load_info = load_pipeline.run(data, table_name="threads")
## Extract, Load, Transform (ELT)
Compared to ETL, ELT moves data once.
-> ingest and store raw data (dlt) -> transform (Hamilton)
+> ingest and store raw data (dlt) -> transform (Apache Hamilton)
Transformations happen within the data destination, typically a data warehouse. To achieve this, we will leverage the [Ibis](https://ibis-project.org/) library, which allows to execute data transformations directly on the destination backend.
1. **Extract & Load**: dlt moves the raw data to the destination
-2. **Transform**: Hamilton + Ibis execute transformations within the destination
+2. **Transform**: Apache Hamilton + Ibis execute transformations within the destination
**Pros**
- Deduplicate computation: redundant operations can be optimized using raw and intermediary data
@@ -204,7 +204,7 @@ Transformations happen within the data destination, typically a data warehouse.
### Transform
-2. Define a dataflow of transformations using Hamilton + Ibis
+2. Define a dataflow of transformations using Apache Hamilton + Ibis
```python
# transform.py
@@ -252,7 +252,7 @@ Transformations happen within the data destination, typically a data warehouse.
return True
```
-3. Execute the Hamilton dataflow to trigger transformations on the backend
+3. Execute the Apache Hamilton dataflow to trigger transformations on the backend
```python
# run.py
@@ -268,7 +268,7 @@ Transformations happen within the data destination, typically a data warehouse.
```
### ELT Summary
-You need to set up your dlt pipeline for raw, and define your Hamilton transformation dataflow. Then, your execution code consist of using dlt to move data to the backend and Hamilton + Ibis to execute transformations.
+You need to set up your dlt pipeline for raw, and define your Apache Hamilton transformation dataflow. Then, your execution code consist of using dlt to move data to the backend and Apache Hamilton + Ibis to execute transformations.
```python
# run.py
@@ -297,7 +297,7 @@ results = dr.execute(
```
## dlt materializer plugin
-We added custom Data Loader/Saver to plug dlt with Hamilton. Compared to the previous approach, it allows to include the dlt operations as part of the Hamilton dataflow and improve lineage / visibility.
+We added custom Data Loader/Saver to plug dlt with Apache Hamilton. Compared to the previous approach, it allows to include the dlt operations as part of the Apache Hamilton dataflow and improve lineage / visibility.
``` {note}
@@ -305,7 +305,7 @@ See [this notebook](https://github.com/apache/hamilton/blob/main/examples/dlt/dl
```
### DataLoader
-The `DataLoader` allows to read in-memory data from a `dlt.Resource`. When working with `dlt.Source`, you can access individual `dlt.Resource` with `source.resource["source_name"]`. This removes the need to write utility functions to read data from dlt (with pandas or Ibis). Contrary to the previous ETL and ELT examples, this approach is useful when you don't want to store the dlt Source data. It effectively connects dlt to Hamilton to enable "Extract, Transform" (ET).
+The `DataLoader` allows to read in-memory data from a `dlt.Resource`. When working with `dlt.Source`, you can access individual `dlt.Resource` with `source.resource["source_name"]`. This removes the need to write utility functions to read data from dlt (with pandas or Ibis). Contrary to the previous ETL and ELT examples, this approach is useful when you don't want to store the dlt Source data. It effectively connects dlt to Apache Hamilton to enable "Extract, Transform" (ET).
```python
@@ -335,7 +335,7 @@ dr.materialize(*materializers, additional_vars=["threads"])
```
### DataSaver
-The `DataSaver` allows to write node results to any `dlt.Destination`. You'll need to define a `dlt.Pipeline` with the desired `dlt.Destination` and you can specify arguments for the `pipeline.run()` behavior (e.g., incremental loading, primary key, load_file_format). This provides a "Transform, Load" (TL) connector from Hamilton to dlt.
+The `DataSaver` allows to write node results to any `dlt.Destination`. You'll need to define a `dlt.Pipeline` with the desired `dlt.Destination` and you can specify arguments for the `pipeline.run()` behavior (e.g., incremental loading, primary key, load_file_format). This provides a "Transform, Load" (TL) connector from Apache Hamilton to dlt.
```python
# run.py
@@ -366,7 +366,7 @@ dr.materialize(*materializers)
```
### Combining both
-You can also combine both the `DataLoader` and `DataSaver`. You will see below that it's almost identical to the ELT example, but now all operations are part of the Hamilton dataflow!
+You can also combine both the `DataLoader` and `DataSaver`. You will see below that it's almost identical to the ELT example, but now all operations are part of the Apache Hamilton dataflow!
```python
diff --git a/docs/integrations/fastapi.md b/docs/integrations/fastapi.md
index 637204cf..1dfb0675 100644
--- a/docs/integrations/fastapi.md
+++ b/docs/integrations/fastapi.md
@@ -22,7 +22,7 @@ if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000) # specify host and port
```
-On this page, you'll learn how Hamilton can help you:
+On this page, you'll learn how Apache Hamilton can help you:
- Test you application
- Reduce the friction from proof-of-concept to production
- Document your API
@@ -35,10 +35,10 @@ FastAPI endpoints are simply decorated Python function, allowing a great deal of
FastAPI already does a great job at automating API documentation by integrating with [Swagger UI](https://fastapi.tiangolo.com/how-to/configure-swagger-ui/) and [OpenAPI](https://fastapi.tiangolo.com/how-to/separate-openapi-schemas/). It leverages the endpoints' name, path, docstring, and type annotations, and also allows to add descriptions and example inputs. However, since docstrings, descriptions, and example inputs are not directly tied to the code, they risk becoming out of sync as [...]
-## Hamilton + FastAPI
-Adding Hamilton to your FastAPI server can provide a better separation between the dataflow and the API endpoints. Each endpoint can use `Driver.execute()` to request variables and wrap results into an HTTP response. Then, data transformations and interactions with resources (e.g., database, web service) are curated into standalone Python modules and decoupled from the server code.
+## Apache Hamilton + FastAPI
+Adding Apache Hamilton to your FastAPI server can provide a better separation between the dataflow and the API endpoints. Each endpoint can use `Driver.execute()` to request variables and wrap results into an HTTP response. Then, data transformations and interactions with resources (e.g., database, web service) are curated into standalone Python modules and decoupled from the server code.
-Since Hamilton dataflows will run the same way inside or outside FastAPI, you can write simpler unit tests for Hamilton functions without defining a mock server and client. Additionnally, visualizations for the defined Hamilton dataflows can be added to the FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs). They will remain in sync with the API behavior because they are generated from the code.
+Since Apache Hamilton dataflows will run the same way inside or outside FastAPI, you can write simpler unit tests for Apache Hamilton functions without defining a mock server and client. Additionnally, visualizations for the defined Apache Hamilton dataflows can be added to the FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs). They will remain in sync with the API behavior because they are generated from the code.
### Example
In this example, we'll build a backend for a PDF summarizer application.
@@ -71,10 +71,10 @@ def post_summarize(
```
> π‘ For more complex FastAPI applications, you can automatically [generate the client code](https://fastapi.tiangolo.com/advanced/generate-clients/) in Python and other languages (TypeScript, Rust, etc.)
-#### Backend dataflow with Hamilton
-Hamilton transformations are defined in the module `summarization.py`. This includes loading and chunking the raw text, summarizing chunks with the OpenAI API, and reducing chunks into a final summary.
+#### Backend dataflow with Apache Hamilton
+Apache Hamilton transformations are defined in the module `summarization.py`. This includes loading and chunking the raw text, summarizing chunks with the OpenAI API, and reducing chunks into a final summary.
-Visualization of the Hamilton dataflow
+Visualization of the Apache Hamilton dataflow
@@ -83,7 +83,7 @@ Then, the FastAPI server is defined in `server.py`. Notice a few things:
- the `Driver` is built only once in the global context.
- the endpoint types are set using `Annotated[...]` to [accept multipart encoded forms](https://fastapi.tiangolo.com/tutorial/request-forms-and-files/?h=form#__tabbed_2_1)
- the HTTP POST request is passed as `inputs` to `Driver.execute()`
-- the Hamilton results are wrapped into a Pydantic `SummarizeResponse` model
+- the Apache Hamilton results are wrapped into a Pydantic `SummarizeResponse` model
```python
# server.py
@@ -97,7 +97,7 @@ import summarization
app = FastAPI()
-# build the Hamilton driver with the summarization module
+# build the Apache Hamilton driver with the summarization module
dr = (
driver.Builder()
.with_modules(summarization)
@@ -133,7 +133,7 @@ if __name__ == "__main__":
```
#### Visualize endpoints' dataflow
-The Hamilton dataflow visualizations can be added to the automatically generated FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs), which can be viewed at http://0.0.0.0:8000/docs
+The Apache Hamilton dataflow visualizations can be added to the automatically generated FastAPI [Swagger UI documentation](https://fastapi.tiangolo.com/features/#automatic-docs), which can be viewed at http://0.0.0.0:8000/docs
```python
# server.py
# ... after defining all endpoints
@@ -154,5 +154,5 @@ app.routes[-1].description += f"""<img src="data:image/png;base64,{base64_viz}"/
### Benefits
- **Separation of concerns**: the decoupling between `server.py` and `summarization.py` makes it easier to extend and test the server separately from the data transformations.
-- **Reusable code**: the module `summarization.py` can be reused elsewhere with Hamilton. For instance, if you first started by building a proof-of-concept with [Streamlit + Hamilton](https://hamilton.dagworks.io/en/latest/integrations/streamlit), the logic you produced could be reused to power your FastAPI server.
-- **Richer documentation**: Hamilton allows to view and better understand the dataflow of an operation. This helps onboard new API users and greatly facilitates transferring the ownership of the API to other engineers.
+- **Reusable code**: the module `summarization.py` can be reused elsewhere with Apache Hamilton. For instance, if you first started by building a proof-of-concept with [Streamlit + Apache Hamilton](https://hamilton.dagworks.io/en/latest/integrations/streamlit), the logic you produced could be reused to power your FastAPI server.
+- **Richer documentation**: Apache Hamilton allows to view and better understand the dataflow of an operation. This helps onboard new API users and greatly facilitates transferring the ownership of the API to other engineers.
diff --git a/docs/integrations/ibis/index.md b/docs/integrations/ibis/index.md
index c6d6ae67..1fb6ae14 100644
--- a/docs/integrations/ibis/index.md
+++ b/docs/integrations/ibis/index.md
@@ -2,7 +2,7 @@
[Ibis](https://ibis-project.org/) is the portable Python dataframe library. It allows you to define data transformations once and execute them in multiple backends (BigQuery, DuckDB, PySpark, SQLite, Snowflake, Postgres, Polars; [see the full list](https://ibis-project.org/support_matrix)). If you never used Ibis before, it should feel similar to SQL with a touch of dataframes (e.g., pandas). You'll be primarily writing expressions (similar to an SQL query), which compute values only aft [...]
-On this page, you'll learn how Ibis + Hamilton can help:
+On this page, you'll learn how Ibis + Apache Hamilton can help:
- Create a modular codebase for better collaboration and maintainability
- Reduce the development-production gap
@@ -47,17 +47,17 @@ Ibis has an SQL-like syntax and supports chaining operations, allowing for power
### Challenge 2 - Orchestrate Ibis code in production
Ibis alleviates a major pain point by enabling data transformations to work across backends. However, moving from dev to prod still requires some code changes such as changing backend connectors, swapping unsupported operators, adding some orchestration and logging execution. This is outside the scope of the Ibis project and is expected to be enabled by other means.
-## How Hamilton complements Ibis
+## How Apache Hamilton complements Ibis
### Write modular Ibis code
-Hamilton was initially developed to [structure pandas code for a large catalog of features](https://blog.dagworks.io/p/tidy-production-pandas-with-hamilton-3b759a2bf562), and has been adopted by multiple organizations since. Its syntax encourages users to chunk code into meaningful and reusable components, which facilitates documentation, unit testing, code reviews, and improves iteration speed. These benefits directly translate to organizing Ibis code.
+Apache Hamilton was initially developed to [structure pandas code for a large catalog of features](https://blog.dagworks.io/p/tidy-production-pandas-with-hamilton-3b759a2bf562), and has been adopted by multiple organizations since. Its syntax encourages users to chunk code into meaningful and reusable components, which facilitates documentation, unit testing, code reviews, and improves iteration speed. These benefits directly translate to organizing Ibis code.
-Now, we'll refactor the above code to use Hamilton. Users have the flexibility to chunk code at the table or the column-level depending on the needed granularity. This modularity is particularly beneficial to Ibis because:
+Now, we'll refactor the above code to use Apache Hamilton. Users have the flexibility to chunk code at the table or the column-level depending on the needed granularity. This modularity is particularly beneficial to Ibis because:
- Well-scoped functions with type annotations and docstring are easier to understand for new Ibis users and facilitate onboarding.
- Unit testing and data validation becomes easier with smaller expressions. These checks become more important when working across backends since the [operation coverage varies](https://ibis-project.org/support_matrix) and bugs may arise.
#### Table-level
-Table-level operations might feel most familiar to SQL and Spark users. Also, Ibis + Hamilton is reminiscent of dbt for the Python ecosystem.
+Table-level operations might feel most familiar to SQL and Spark users. Also, Ibis + Apache Hamilton is reminiscent of dbt for the Python ecosystem.
Working with tables is very efficient when your number of columns/features is limited, and you don't need full lineage. As you want to reuse components, you can progressively breakdown "table-level code" in to "column-level code".
@@ -97,7 +97,7 @@ def feature_set(
#### Column-level
-Hamilton was built around column-level operations, which is most common in dataframe libraries (pandas, Dask, polars).
+Apache Hamilton was built around column-level operations, which is most common in dataframe libraries (pandas, Dask, polars).
Column-level code leads to fully-reusable feature definitions and a great level of lineage. Notably, this allows to [trace sensitive data and evaluate downstream impacts of code changes](../../how-tos/use-hamilton-for-lineage.md). However, it is more verbose to get started with, but remember that code is more often read than written.
@@ -162,13 +162,13 @@ def feature_set(
``` {note}
-If your code is already structured with Hamilton, migrating from pandas to Ibis should be easy!
+If your code is already structured with Apache Hamilton, migrating from pandas to Ibis should be easy!
```
### Orchestrate Ibis anywhere
-Hamilton is ideal orchestrate for Ibis because it has the lightest footprint and will run anywhere Python does (script, notebook, FastAPI, pyodide, etc.) In fact, the Hamilton library only has 4 dependencies. You don't need "framework code" to get started, just plain Python functions. When moving to production, Hamilton has all the necessary features to complement Ibis such as swapping components, configurations, and lifecycle hooks for logging, alerting, and telemetry.
+Apache Hamilton is ideal orchestrate for Ibis because it has the lightest footprint and will run anywhere Python does (script, notebook, FastAPI, pyodide, etc.) In fact, the Apache Hamilton library only has 4 dependencies. You don't need "framework code" to get started, just plain Python functions. When moving to production, Apache Hamilton has all the necessary features to complement Ibis such as swapping components, configurations, and lifecycle hooks for logging, alerting, and telemetry.
-A simple usage pattern of Hamilton + Ibis is to use the [`@config.when` function modifier](../../concepts/function-modifiers.rst). In the following example, we have alternative implementations for the backend connection, which will be used for computing and storing results. When running your code, specify in your config `backend="duckdb"` or `backend="bigquery"` to swap between the two.
+A simple usage pattern of Apache Hamilton + Ibis is to use the [`@config.when` function modifier](../../concepts/function-modifiers.rst). In the following example, we have alternative implementations for the backend connection, which will be used for computing and storing results. When running your code, specify in your config `backend="duckdb"` or `backend="bigquery"` to swap between the two.
```python
# ibis_dataflow.py
@@ -210,20 +210,20 @@ def insert_results(
)
```
``` {note}
-A potential architecture for Ibis + Hamilton would be running CRON jobs on GitHub actions to periodically launch AWS Lambda with the Hamilton code to orchestrate Ibis data transformations directly on the backend. This has the potential to save meaningful cloud egress cost and greatly diminishes orchestration complexity.
+A potential architecture for Ibis + Apache Hamilton would be running CRON jobs on GitHub actions to periodically launch AWS Lambda with the Apache Hamilton code to orchestrate Ibis data transformations directly on the backend. This has the potential to save meaningful cloud egress cost and greatly diminishes orchestration complexity.
```
-## How Ibis complements Hamilton
+## How Ibis complements Apache Hamilton
### Performance boost
-Leveraging DuckDB as the default backend, Hamilton users migrating to Ibis should immediately find performance improvements both for local dev and production. In addition, the portability of Ibis has the potential to greatly reduce development time.
+Leveraging DuckDB as the default backend, Apache Hamilton users migrating to Ibis should immediately find performance improvements both for local dev and production. In addition, the portability of Ibis has the potential to greatly reduce development time.
### Atomic data transformation documentation
-Hamilton can directly produce a dataflow visualization from code, helping with project documentation. Ibis pushes this one step further by providing a detailed view of the query plan and schemas. See this Ibis visualization for the column-level Hamilton dataflow defined above. It includes all renaming, type casting, and transformations steps (Please open the image in a new tab and zoom in π).
+Apache Hamilton can directly produce a dataflow visualization from code, helping with project documentation. Ibis pushes this one step further by providing a detailed view of the query plan and schemas. See this Ibis visualization for the column-level Apache Hamilton dataflow defined above. It includes all renaming, type casting, and transformations steps (Please open the image in a new tab and zoom in π).
### Working across rows with user-defined functions (UDFs)
-Hamilton and most backends are designed to work primarily on tables and columns, but sometimes you'd like to operate over a row (think of `pd.DataFrame.apply()`). However, pivoting tables is costly and manually iterating over rows to collect values and create a new column is quickly inconvenient. By [using scalar user-defined functions (UDFs)](https://ibis-project.org/reference/scalar-udfs), Ibis makes it possible to execute arbitrary Python code on rows directly on the backend.
+Apache Hamilton and most backends are designed to work primarily on tables and columns, but sometimes you'd like to operate over a row (think of `pd.DataFrame.apply()`). However, pivoting tables is costly and manually iterating over rows to collect values and create a new column is quickly inconvenient. By [using scalar user-defined functions (UDFs)](https://ibis-project.org/reference/scalar-udfs), Ibis makes it possible to execute arbitrary Python code on rows directly on the backend.
``` {note}
Using `@ibis.udf.scalar.python` creates a non-vectorized function that iterates row-by-row. See [the docs](https://ibis-project.org/reference/scalar-udfs) to use backend-specific UDFs with `@ibis.udf.scalar.builtin` and create vectorized scalar UDFs.
@@ -275,7 +275,7 @@ def summaries(documents: ir.Table, prompt_template: str) -> ir.Table
-## Ibis + Hamilton - a natural pairing
-- **What works in dev works in prod**: Ibis and Hamilton allows you to write and structure code data transformations portable across backends for small and big data alike. The two being lightweight libraries, installing dependencies on remote workers is fast and you're unlikely to ever encounter dependency conflicts.
+## Ibis + Apache Hamilton - a natural pairing
+- **What works in dev works in prod**: Ibis and Apache Hamilton allows you to write and structure code data transformations portable across backends for small and big data alike. The two being lightweight libraries, installing dependencies on remote workers is fast and you're unlikely to ever encounter dependency conflicts.
- **Maintainable and testable code**: Modular functions facilitates writing high quality code and promotes reusability, compounding your engineering efforts. It becomes easier for new users to contribute to a dataflow and pull requests are merged faster.
-- **Greater visibility**: With Hamilton and Ibis, you have incredible visualizations directly derived from your code. This is a superpower for documentation, allowing users to make sense of a dataflow, and also reason about changes.
+- **Greater visibility**: With Apache Hamilton and Ibis, you have incredible visualizations directly derived from your code. This is a superpower for documentation, allowing users to make sense of a dataflow, and also reason about changes.
diff --git a/docs/integrations/index.rst b/docs/integrations/index.rst
index 09d448f4..e131dd8b 100644
--- a/docs/integrations/index.rst
+++ b/docs/integrations/index.rst
@@ -2,7 +2,7 @@
Integrations
============
-This section showcases how Hamilton integrates with popular frameworks.
+This section showcases how Apache Hamilton integrates with popular frameworks.
.. toctree::
dlt/index
diff --git a/docs/integrations/streamlit.md b/docs/integrations/streamlit.md
index 8a272ec5..db391b28 100644
--- a/docs/integrations/streamlit.md
+++ b/docs/integrations/streamlit.md
@@ -1,6 +1,6 @@
# Streamlit
-[Streamlit](https://streamlit.io/) is an open-source Python library to create web applications with minimal effort. It's an effective solution to create simple dashboards, interactive data visualizations, and proof-of-concepts for data science, machine learning, and LLM applications. On this page, you'll learn how Hamilton can help you:
+[Streamlit](https://streamlit.io/) is an open-source Python library to create web applications with minimal effort. It's an effective solution to create simple dashboards, interactive data visualizations, and proof-of-concepts for data science, machine learning, and LLM applications. On this page, you'll learn how Apache Hamilton can help you:
- Write cleaner Streamlit applications
- Reduce friction transition between proof-of-concept and production
- Improve Streamlit performance
@@ -18,7 +18,7 @@ def greeting(name: str) -> None:
st.write(f"Hello {name}")
def app():
- st.title("Hamilton + Streamlit π±βπ")
+ st.title("Apache Hamilton + Streamlit π±βπ")
main, settings = st.tabs(["Main", "Settings"])
left, right = st.columns(2)
@@ -50,7 +50,7 @@ def load_dataframe(path: str) -> pd.DataFrame:
return pd.read_parquet(path)
def app():
- st.title("Hamilton + Streamlit π±βπ")
+ st.title("Apache Hamilton + Streamlit π±βπ")
# load_dataframe() will only run the first time
df = load_dataframe(path="...")
@@ -72,15 +72,15 @@ if __name__ == "__main__":
> β This example is illustratory and real applications quickly get more complex.
-## Hamilton + Streamlit
-Adding Hamilton to your Streamlit application can provide a better separation between the dataflow and the UI logic. They pair nicely together because Hamilton is also stateless. Once defined, each call to `Driver.execute()` is independent. Therefore, on each Streamlit rerun, you use `Driver.execute()` to complete computations. Using Hamilton this way allows you to write your dataflow into Python modules and outside of the Streamlit.
+## Apache Hamilton + Streamlit
+Adding Apache Hamilton to your Streamlit application can provide a better separation between the dataflow and the UI logic. They pair nicely together because Apache Hamilton is also stateless. Once defined, each call to `Driver.execute()` is independent. Therefore, on each Streamlit rerun, you use `Driver.execute()` to complete computations. Using Apache Hamilton this way allows you to write your dataflow into Python modules and outside of the Streamlit.
### Example
In this example, we will build a simple financial dashboard based on the Kaggle [Bank Marketing Dataset](https://www.kaggle.com/datasets/janiobachmann/bank-marketing-dataset).
> The full code can be found on [GitHub](https://github.com/apache/hamilton/tree/main/examples/streamlit)
-First, Hamilton transformations are defined in the module `logic.py`. This includes downloading the data from the web, getting unique values for `job`, conducting groupby aggregates, and creating `plotly` figures.
+First, Apache Hamilton transformations are defined in the module `logic.py`. This includes downloading the data from the web, getting unique values for `job`, conducting groupby aggregates, and creating `plotly` figures.
```python
# logic.py
import pandas as pd
@@ -118,7 +118,7 @@ def job_hist(job_df: pd.DataFrame) -> Figure:
Then, the Streamlit UI is defined in `app.py`. Notice a few things:
- `app.py` doesn't have to depend on `pandas` and `plotly`.
- `@cache_resource` allows to create the `Driver` only once.
-- `@cache_data` on `_execute()` will automatically cache any Hamilton result based on the combination of arguments (`final_vars`, `inputs`, and `overrides`)
+- `@cache_data` on `_execute()` will automatically cache any Apache Hamilton result based on the combination of arguments (`final_vars`, `inputs`, and `overrides`)
- `get_state_inputs()` and `get_state_overrides()` will collect values from user inputs.
- `execute()` parses the inputs and overrides from the state and call `_execute()`.
@@ -147,7 +147,7 @@ def _execute(
inputs: Optional[dict] = None,
overrides: Optional[dict] = None,
) -> dict:
- """Generic utility to cache Hamilton results"""
+ """Generic utility to cache Apache Hamilton results"""
dr = get_hamilton_driver()
return dr.execute(final_vars, inputs=inputs, overrides=overrides)
@@ -167,7 +167,7 @@ def execute(final_vars: list[str]):
def app():
- st.title("Hamilton + Streamlit π±βπ")
+ st.title("Apache Hamilton + Streamlit π±βπ")
# run the base data that always needs to be displayed
data = execute(["all_jobs", "balance_per_job", "balance_per_job_boxplot"])
@@ -189,7 +189,7 @@ if __name__ == "__main__":
### Benefits
- **Clearer scope**: the decoupling between `app.py` and `logic.py` makes it easier to add data transformations or extend UI, and debug errors associated with either.
-- **Reusable code**: the module `logic.py` can be reused elsewhere with Hamilton.
- - If you are building a proof-of-concept with Streamlit, your Hamilton module will be able to grow with your project and be useful for your production pipelines.
- - If you are already building dataflows with Hamilton, using it with Streamlit ensures your dashboard metrics have the same implementation with your production pipeline (i.e., prevent [implementation skew](https://building.nubank.com.br/dealing-with-train-serve-skew-in-real-time-ml-models-a-short-guide/))
-- **Performance boost**: by caching the Hamilton Driver and its execution call, we are able to effectively cache all data operations in a few lines of code. Furthermore, Hamilton can scale further by using a remote task executor on a separate machine from the Streamlit application.
+- **Reusable code**: the module `logic.py` can be reused elsewhere with Apache Hamilton.
+ - If you are building a proof-of-concept with Streamlit, your Apache Hamilton module will be able to grow with your project and be useful for your production pipelines.
+ - If you are already building dataflows with Apache Hamilton, using it with Streamlit ensures your dashboard metrics have the same implementation with your production pipeline (i.e., prevent [implementation skew](https://building.nubank.com.br/dealing-with-train-serve-skew-in-real-time-ml-models-a-short-guide/))
+- **Performance boost**: by caching the Apache Hamilton Driver and its execution call, we are able to effectively cache all data operations in a few lines of code. Furthermore, Apache Hamilton can scale further by using a remote task executor on a separate machine from the Streamlit application.
diff --git a/docs/main.md b/docs/main.md
index e2d5cdcb..f4b8d20d 100644
--- a/docs/main.md
+++ b/docs/main.md
@@ -1,34 +1,34 @@
-# Welcome to Hamilton
+# Welcome to Apache Hamilton
<div align="left">
- <a href="https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g" target="_blank"><img src="https://img.shields.io/badge/Join-Hamilton_Slack-brightgreen?logo=slack" alt="Hamilton Slack"/></a>
+ <a href="https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g" target="_blank"><img src="https://img.shields.io/badge/Join-Hamilton_Slack-brightgreen?logo=slack" alt="Apache Hamilton Slack"/></a>
<a href="https://twitter.com/hamilton_os" target="_blank"><img src="https://img.shields.io/twitter/url/http/shields.io.svg?style=social" alt="Twitter"/></a>
<a href="https://pepy.tech/project/sf-hamilton" target="_blank"><img src="https://pepy.tech/badge/sf-hamilton" alt="Total Downloads"/></a>
<a href="https://pepy.tech/project/sf-hamilton" target="_blank"><img src="https://static.pepy.tech/badge/sf-hamilton/month" alt="Total Monthly Downloads"/></a>
</div>
-Hamilton is a general-purpose framework to write dataflows using regular Python functions. At the core, each function defines a transformation and its parameters indicates its dependencies. Hamilton automatically connects individual functions into a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) that can be executed, visualized, optimized, and reported on.
-Hamilton also comes with a [UI](hamilton-ui/index.rst) to visualize, catalog, and monitor your dataflows.
+Apache Hamilton is a general-purpose framework to write dataflows using regular Python functions. At the core, each function defines a transformation and its parameters indicates its dependencies. Apache Hamilton automatically connects individual functions into a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) that can be executed, visualized, optimized, and reported on.
+Apache Hamilton also comes with a [UI](hamilton-ui/index.rst) to visualize, catalog, and monitor your dataflows.
```{figure} ./_static/abc.png
:scale: 50
:align: center
-The ABC of Hamilton
+The ABC of Apache Hamilton
```
-# Why should you use Hamilton?
-**Facilitate collaboration**. By focusing on functions, Hamilton avoids sprawling code hierarchy and generates flat dataflows. Well-scoped functions make it easier to add features, complete code reviews, debug pipeline failures, and hand-off projects. Visualizations can be generated directly from your code to better understand and document it.
-Integration with the [Hamilton UI](hamilton-ui/index.rst) allows you to track lineage, catalog code & artifacts, and monitor your dataflows.
+# Why should you use Apache Hamilton?
+**Facilitate collaboration**. By focusing on functions, Apache Hamilton avoids sprawling code hierarchy and generates flat dataflows. Well-scoped functions make it easier to add features, complete code reviews, debug pipeline failures, and hand-off projects. Visualizations can be generated directly from your code to better understand and document it.
+Integration with the [Apache Hamilton UI](hamilton-ui/index.rst) allows you to track lineage, catalog code & artifacts, and monitor your dataflows.
-**Reduce development time**. Hamilton dataflows are reusable across projects and context (e.g., pipeline vs. web service). The benefits of developing robust and well-tested solutions are multiplied by reusability. Off-the-shelf dataflows are available on the [Hamilton Hub](https://hub.dagworks.io/).
+**Reduce development time**. Apache Hamilton dataflows are reusable across projects and context (e.g., pipeline vs. web service). The benefits of developing robust and well-tested solutions are multiplied by reusability. Off-the-shelf dataflows are available on the [Apache Hamilton Hub](https://hub.dagworks.io/).
-**Own your platform**. Hamilton helps you integrate the frameworks and tools of your stack. Hamilton's features are easy to extend and customize to your needs. This flexibility enables self-serve designs and ultimately reduces the risks of vendor lock-in.
+**Own your platform**. Apache Hamilton helps you integrate the frameworks and tools of your stack. Apache Hamilton's features are easy to extend and customize to your needs. This flexibility enables self-serve designs and ultimately reduces the risks of vendor lock-in.
-**Scale your dataflow**. Hamilton separates transformation logic from execution, allowing you to seamlessly scale via remote execution (AWS, Modal, etc.) and specialized computation engines (Spark, Ray, duckdb etc.). Hamilton was battle tested under intensive enterprise data workloads.
+**Scale your dataflow**. Apache Hamilton separates transformation logic from execution, allowing you to seamlessly scale via remote execution (AWS, Modal, etc.) and specialized computation engines (Spark, Ray, duckdb etc.). Apache Hamilton was battle tested under intensive enterprise data workloads.
-Here's a quick overview of benefits that Hamilton provides as compared to other tools:
+Here's a quick overview of benefits that Apache Hamilton provides as compared to other tools:
-| Feature | Hamilton | Macro orchestration systems (e.g. Airflow) | Feast | dbt | Dask |
+| Feature | Apache Hamilton | Macro orchestration systems (e.g. Airflow) | Feast | dbt | Dask |
|-------------------------------------------|:---:|:---------------------------------------------:|:-----:|:---:|:----:|
| Execute a graph of data transformations | β
| β
| β | β
| β
|
| Can visualize lineage easily | β
| β | β | β
| β
|
@@ -42,11 +42,11 @@ Here's a quick overview of benefits that Hamilton provides as compared to other
# Architecture Overview
-The following diagram gives a simplified overview of the main components of Hamilton.
+The following diagram gives a simplified overview of the main components of Apache Hamilton.
-**Functions & Module**. Transformations are regular Python functions organized into modules. Functions must be type-annotated, but hold no dependency with Hamilton and can be reused outside of it.
+**Functions & Module**. Transformations are regular Python functions organized into modules. Functions must be type-annotated, but hold no dependency with Apache Hamilton and can be reused outside of it.
**Driver & FunctionGraph**. The `Driver` will automatically assemble the `FunctionGraph` from the modules given. The `Driver` can be configured to modify and extend the execution behavior (e.g., remote execution, monitoring, webhooks, caching).
@@ -54,10 +54,10 @@ The following diagram gives a simplified overview of the main components of Hami
**Execution**. When requesting variables, the `Driver` establishes an execution plan to only compute the required functions. Then, results are gathered and returned to the user.
-# Who is using Hamilton?
+# Who is using Apache Hamilton?
-Multiple companies are doing cool stuff with Hamilton! Come chat with members of the community and the development team on [Slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g):
+Multiple companies are doing cool stuff with Apache Hamilton! Come chat with members of the community and the development team on [Slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g):
* **Wealth.com** - Async Python LLM document processing pipelines
* **Wren.ai** - Async RAG pipelines
@@ -89,28 +89,28 @@ Multiple companies are doing cool stuff with Hamilton! Come chat with members of
<div class="testimonial-container">
<div class="testimonial-card">
<div class="testimonial-content">
- <p>"Hamilton provides a modular and compatible framework that has significantly empowered our data science team. We've been able to build robust and flexible data pipelines with ease. The documentation is thorough and regularly updated... Even with no prior experience with the package, our team successfully migrated one of our legacy data pipelines to the Hamilton structure within a month. This transition has greatly enhanced our productivity, enabling us to focus more on [...]
+ <p>"Apache Hamilton provides a modular and compatible framework that has significantly empowered our data science team. We've been able to build robust and flexible data pipelines with ease. The documentation is thorough and regularly updated... Even with no prior experience with the package, our team successfully migrated one of our legacy data pipelines to the Apache Hamilton structure within a month. This transition has greatly enhanced our productivity, enabling us to [...]
<h4>Yuan Liu</h4>
<span>DS, Kora Financial</span>
</div>
</div>
<div class="testimonial-card">
<div class="testimonial-content">
- <p>"How (with good software practices) do you orchestrate a system of asynchronous LLM calls, but where some of them depend on others? How do you build such a system so that itβs modular and testable? At wealth.com we've selected Hamilton to help us solve these problems and others. And today our product, Ester AI, an AI legal assistant that extracts information from estate planning documents, is running in production with Hamilton under the hood."</p>
+ <p>"How (with good software practices) do you orchestrate a system of asynchronous LLM calls, but where some of them depend on others? How do you build such a system so that itβs modular and testable? At wealth.com we've selected Apache Hamilton to help us solve these problems and others. And today our product, Ester AI, an AI legal assistant that extracts information from estate planning documents, is running in production with Apache Hamilton under the hood."</p>
<h4>Kyle Pounder</h4>
<span>CTO, Wealth.com</span>
</div>
</div>
<div class="testimonial-card">
<div class="testimonial-content">
- <p>"Hamilton is simplicity. Its declarative approach to defining pipelines (as well as the UI to visualize them) makes testing and modifying the code easy, and onboarding is quick and painless. Since using Hamilton, we have improved our efficiency of both developing new functionality and onboarding new developers to work on the code. We deliver solutions more quickly than before."</p>
+ <p>"Apache Hamilton is simplicity. Its declarative approach to defining pipelines (as well as the UI to visualize them) makes testing and modifying the code easy, and onboarding is quick and painless. Since using Apache Hamilton, we have improved our efficiency of both developing new functionality and onboarding new developers to work on the code. We deliver solutions more quickly than before."</p>
<h4>MichaΕ Siedlaczek</h4>
<span>Senior DS/SWE, IBM</span>
</div>
</div>
<div class="testimonial-card">
<div class="testimonial-content">
- <p>"...The companion Hamilton UI has taken the value proposition up enormously with the ability to clearly show lineage & track execution times, covering a major part of our observability needs"</p>
+ <p>"...The companion Apache Hamilton UI has taken the value proposition up enormously with the ability to clearly show lineage & track execution times, covering a major part of our observability needs"</p>
<h4>Fran Boon</h4>
<span>Director, Oxehealth.com</span>
</div>
diff --git a/docs/reference/caching/caching-logic.rst b/docs/reference/caching/caching-logic.rst
index 94e827cc..b467add6 100644
--- a/docs/reference/caching/caching-logic.rst
+++ b/docs/reference/caching/caching-logic.rst
@@ -55,12 +55,12 @@ Quirks and limitations
Caching is a large and complex feature. This section is an attempt to list quirks and limitations, known and theoretical, to help debugging and guide feature development
-- The standard library includes a lot of types which are not primitives. Thus, Hamilton might not be supporting them explicitly. It should be simple to add, so ping us if you need it.
+- The standard library includes a lot of types which are not primitives. Thus, Apache Hamilton might not be supporting them explicitly. It should be simple to add, so ping us if you need it.
- The ``ResultStore`` could be architectured better to support custom formats. Right now, we use a ``DataSaver`` to produce the ``.parquet`` file and we pickle the ``DataLoader`` for later retrieval. Then, the metadata and result stores are completely unaware of the ``.parquet`` file making it difficult to handle cache eviction.
- When a function with default parameter values passes through lifecycle hooks, the default values are not part of the ``node_kwargs``. They need to be retrieved manually from the ``node.Node`` object.
-- supporting the Hamilton ``AsyncDriver`` would require making the adapter async, but also the stores. A potential challenge is ensuring that you can use the same cache (i.e., same SQLite db and filesystem) for both sync and async drivers.
+- supporting the Apache Hamilton ``AsyncDriver`` would require making the adapter async, but also the stores. A potential challenge is ensuring that you can use the same cache (i.e., same SQLite db and filesystem) for both sync and async drivers.
- If the ``@cache`` allows to specify the ``format`` (e.g., ``json``, ``parquet``), we probably want ``.with_cache()`` to support the same feature.
-- Hamilton allows a single ``do_node_execute()`` hook. Since the caching feature uses it, it is currently incompatible with other adapters leveraging it (``PDBDebugger``, ``CacheAdapter`` (deprecated), ``GracefulErrorAdapter`` (somewhat redundant with caching), ``DiskCacheAdapter`` (deprecated), ``NarwhalsAdapter`` (could be refactored))
+- Apache Hamilton allows a single ``do_node_execute()`` hook. Since the caching feature uses it, it is currently incompatible with other adapters leveraging it (``PDBDebugger``, ``CacheAdapter`` (deprecated), ``GracefulErrorAdapter`` (somewhat redundant with caching), ``DiskCacheAdapter`` (deprecated), ``NarwhalsAdapter`` (could be refactored))
- the presence of MD5 hashing can be seen as a security risk and prevent adoption. `read more in DVC issues <https://github.com/iterative/dvc/issues/3069>`_
- when hitting the base case of ``fingerprinting.hash_value()`` we return the constant ``UNHASHABLE_VALUE``. If the adapter receives this value, it will append a random UUID to it. This is to prevent collision between unhashable types. This ``data_version`` is no longer deterministic, but the value can still be retrieved or be part of another node's ``cache_key``.
- having ``@functools.singledispatch(object)`` allows to override the base case of ``hash_value()`` because it will catch all types.
diff --git a/docs/reference/dataflows/index.rst b/docs/reference/dataflows/index.rst
index 21efc592..5dc4871a 100644
--- a/docs/reference/dataflows/index.rst
+++ b/docs/reference/dataflows/index.rst
@@ -4,7 +4,7 @@ Dataflows
Here lies reference documentation for `dataflows` module functions that
integrate with the `hub.dagworks.io <https://hub.dagworks.io>`_ so you can pull off-the-shelf dataflows
-and get started quickly with Hamilton.
+and get started quickly with Apache Hamilton.
Reference
---------
diff --git a/docs/reference/dataflows/inspect.rst b/docs/reference/dataflows/inspect.rst
index ae2e7e40..0a7329ce 100644
--- a/docs/reference/dataflows/inspect.rst
+++ b/docs/reference/dataflows/inspect.rst
@@ -1,7 +1,7 @@
=======================
inspect()
=======================
-Use this to get cursory information about a Hamilton module.
+Use this to get cursory information about a Apache Hamilton module.
.. autoclass:: hamilton.dataflows.InspectResult
diff --git a/docs/reference/dataflows/inspect_module.rst b/docs/reference/dataflows/inspect_module.rst
index 2e8c587c..ef1ae21a 100644
--- a/docs/reference/dataflows/inspect_module.rst
+++ b/docs/reference/dataflows/inspect_module.rst
@@ -1,7 +1,7 @@
=======================
inspect_module()
=======================
-Use this to get deep information about a Hamilton module.
+Use this to get deep information about a Apache Hamilton module.
.. autoclass:: hamilton.dataflows.InspectModuleResult
diff --git a/docs/reference/decorators/index.rst b/docs/reference/decorators/index.rst
index 204159de..d1570227 100644
--- a/docs/reference/decorators/index.rst
+++ b/docs/reference/decorators/index.rst
@@ -2,7 +2,7 @@
Decorators
===========
-Hamilton implements several decorators to promote business-logic deduplication, configuratibility, and add a layer of capabilities. These decorators can be found in the ``hamilton.function_modifiers`` submodule `GitHub <https://github.com/apache/hamilton/blob/main/hamilton/function_modifiers>`__.
+Apache Hamilton implements several decorators to promote business-logic deduplication, configuratibility, and add a layer of capabilities. These decorators can be found in the ``hamilton.function_modifiers`` submodule `GitHub <https://github.com/apache/hamilton/blob/main/hamilton/function_modifiers>`__.
Custom Decorators
-----------------
diff --git a/docs/reference/disabling-telemetry.md b/docs/reference/disabling-telemetry.md
index 5cce4d88..03e9d4f6 100644
--- a/docs/reference/disabling-telemetry.md
+++ b/docs/reference/disabling-telemetry.md
@@ -1,7 +1,7 @@
# Telemetry
If you do not wish to participate in telemetry capture, one can opt-out with one of the following methods:
-1. Set it to false programmatically in your code before creating a Hamilton driver:
+1. Set it to false programmatically in your code before creating a Apache Hamilton driver:
```python
from hamilton import telemetry
telemetry.disable_telemetry()
diff --git a/docs/reference/drivers/index.rst b/docs/reference/drivers/index.rst
index 0d01b07b..491df069 100644
--- a/docs/reference/drivers/index.rst
+++ b/docs/reference/drivers/index.rst
@@ -10,7 +10,7 @@ It's highly parameterizable, allowing you to customize:
To tune the above, pass in a Graph Adapter, a Result Builder, and/or anotehr lifecycle method -- see :doc:`../result-builders/index`, :doc:`../graph-adapters/index`.
-Let's walk through how you might use the Hamilton Driver.
+Let's walk through how you might use the Apache Hamilton Driver.
Instantiation
=============
@@ -66,7 +66,7 @@ and instead you provide that input to each ``execute`` invocation.
Short circuiting some DAG computation
*************************************
-This will force Hamilton to short circuit a particular computation path, and use the passed in override as a result of
+This will force Apache Hamilton to short circuit a particular computation path, and use the passed in override as a result of
that particular node.
diff --git a/docs/reference/graph-adapters/DaskGraphAdapter.rst b/docs/reference/graph-adapters/DaskGraphAdapter.rst
index d22f1ec4..e51a73a1 100644
--- a/docs/reference/graph-adapters/DaskGraphAdapter.rst
+++ b/docs/reference/graph-adapters/DaskGraphAdapter.rst
@@ -2,7 +2,7 @@
h_dask.DaskGraphAdapter
=======================
-Runs the entire Hamilton DAG on dask.
+Runs the entire Apache Hamilton DAG on dask.
.. autoclass:: hamilton.plugins.h_dask.DaskGraphAdapter
diff --git a/docs/reference/graph-adapters/RayGraphAdapter.rst b/docs/reference/graph-adapters/RayGraphAdapter.rst
index f3aa6e88..aa69f5df 100644
--- a/docs/reference/graph-adapters/RayGraphAdapter.rst
+++ b/docs/reference/graph-adapters/RayGraphAdapter.rst
@@ -2,7 +2,7 @@
h_ray.RayGraphAdapter
=======================
-The graph adapter to delegate execution of the individual nodes in a Hamilton graph to Ray.
+The graph adapter to delegate execution of the individual nodes in a Apache Hamilton graph to Ray.
.. autoclass:: hamilton.plugins.h_ray.RayGraphAdapter
:special-members: __init__
diff --git a/docs/reference/graph-adapters/ThreadPoolFutureAdapter.rst b/docs/reference/graph-adapters/ThreadPoolFutureAdapter.rst
index ca34d146..9b73d013 100644
--- a/docs/reference/graph-adapters/ThreadPoolFutureAdapter.rst
+++ b/docs/reference/graph-adapters/ThreadPoolFutureAdapter.rst
@@ -2,7 +2,7 @@
h_threadpool.FutureAdapter
==========================
-This is an adapter to delegate execution of the individual nodes in a Hamilton graph to a threadpool.
+This is an adapter to delegate execution of the individual nodes in a Apache Hamilton graph to a threadpool.
This is useful when you have a graph with many nodes that can be executed in parallel.
.. autoclass:: hamilton.plugins.h_threadpool.FutureAdapter
diff --git a/docs/reference/graph-adapters/index.rst b/docs/reference/graph-adapters/index.rst
index ffb43dd6..08c94575 100644
--- a/docs/reference/graph-adapters/index.rst
+++ b/docs/reference/graph-adapters/index.rst
@@ -2,7 +2,7 @@
GraphAdapters
==============
-This section helps determine ways to execute Hamilton.
+This section helps determine ways to execute Apache Hamilton.
Note that these are special cases of the :doc:`../lifecycle-hooks/index` meant to help with execution. They implement multiple lifecycle customizations in a single place.
Reference
diff --git a/docs/reference/io/index.rst b/docs/reference/io/index.rst
index 25c563c7..cf8ba196 100644
--- a/docs/reference/io/index.rst
+++ b/docs/reference/io/index.rst
@@ -2,7 +2,7 @@
I/O
==============
-This section contains any information about I/O within Hamilton. If you're using materializers
+This section contains any information about I/O within Apache Hamilton. If you're using materializers
or the `save_to`/`load_from` decorator, you'll need this page to help you find the
set of available loading/saving targets.
diff --git a/docs/reference/lifecycle-hooks/ProgressBar.rst b/docs/reference/lifecycle-hooks/ProgressBar.rst
index 280ded72..9e58d56e 100644
--- a/docs/reference/lifecycle-hooks/ProgressBar.rst
+++ b/docs/reference/lifecycle-hooks/ProgressBar.rst
@@ -2,7 +2,7 @@
plugins.h_tqdm.ProgressBar
==========================
-Provides a progress bar for Hamilton execution. Must have `tqdm` installed to use it:
+Provides a progress bar for Apache Hamilton execution. Must have `tqdm` installed to use it:
`pip install sf-hamilton[tqdm]` (use quotes if using zsh)
diff --git a/docs/reference/lifecycle-hooks/RichProgressBar.rst b/docs/reference/lifecycle-hooks/RichProgressBar.rst
index 0a24b0c7..9390e547 100644
--- a/docs/reference/lifecycle-hooks/RichProgressBar.rst
+++ b/docs/reference/lifecycle-hooks/RichProgressBar.rst
@@ -2,7 +2,7 @@
plugins.h_rich.RichProgressBar
==============================
-Provides a progress bar for Hamilton execution. Must have `rich` installed to use it:
+Provides a progress bar for Apache Hamilton execution. Must have `rich` installed to use it:
`pip install sf-hamilton[rich]` (use quotes if using zsh)
diff --git a/docs/reference/lifecycle-hooks/SlackNotifierHook.rst b/docs/reference/lifecycle-hooks/SlackNotifierHook.rst
index ff3d1e8b..dfcbbd5f 100644
--- a/docs/reference/lifecycle-hooks/SlackNotifierHook.rst
+++ b/docs/reference/lifecycle-hooks/SlackNotifierHook.rst
@@ -2,7 +2,7 @@
plugins.h_slack.SlackNotifier
=============================
-Provides a Slack notifier for Hamilton execution. Must have `slack_sdk` installed to use it:
+Provides a Slack notifier for Apache Hamilton execution. Must have `slack_sdk` installed to use it:
`pip install sf-hamilton[slack]` (use quotes if using zsh)
diff --git a/docs/reference/lifecycle-hooks/index.rst b/docs/reference/lifecycle-hooks/index.rst
index 9817fdc7..24b1eaaa 100644
--- a/docs/reference/lifecycle-hooks/index.rst
+++ b/docs/reference/lifecycle-hooks/index.rst
@@ -9,7 +9,7 @@ The other extensions live within `plugins <https://github.com/apache/hamilton/tr
Customization
-------------
-The subsequent documents contain public-facing APIs for customizing Hamilton's
+The subsequent documents contain public-facing APIs for customizing Apache Hamilton's
execution. Note that the public-facing APIs are still a work in progress -- we
will be improving the documentation. We plan for the APIs, however, to be stable
looking forward.
diff --git a/examples/LLM_Workflows/GraphRAG/README.md b/examples/LLM_Workflows/GraphRAG/README.md
index d09dbff6..c6b5a196 100644
--- a/examples/LLM_Workflows/GraphRAG/README.md
+++ b/examples/LLM_Workflows/GraphRAG/README.md
@@ -83,7 +83,7 @@ This will run the following two pipelines:
-**Note:** [Hamilton](https://github.com/apache/hamilton) also comes with a UI that you can use to visualize the pipeline and
+**Note:** [Apache Hamilton](https://github.com/apache/hamilton) also comes with a UI that you can use to visualize the pipeline and
track execution information about it. See hamilton_ingest.py or ingest_notebook.ipynb for more information.
## Ingest data using a notebook:
diff --git a/examples/LLM_Workflows/RAG_document_extract_chunk_embed/README.md b/examples/LLM_Workflows/RAG_document_extract_chunk_embed/README.md
index ffe7cec7..05b189a0 100644
--- a/examples/LLM_Workflows/RAG_document_extract_chunk_embed/README.md
+++ b/examples/LLM_Workflows/RAG_document_extract_chunk_embed/README.md
@@ -2,7 +2,7 @@
-The notebook here explains how to build a simple pipeline that processes a page from Hamilton's documentation.
+The notebook here explains how to build a simple pipeline that processes a page from Apache Hamilton's documentation.
Open it in google collab:
<a target="_blank" href="https://colab.research.google.com/github/DAGWorks-Inc/hamilton/blob/main/examples/LLM_Workflows/RAG_document_extract_chunk_embed/simple_pipeline.ipynb">
diff --git a/examples/LLM_Workflows/README.md b/examples/LLM_Workflows/README.md
index a0ab8688..038041c0 100644
--- a/examples/LLM_Workflows/README.md
+++ b/examples/LLM_Workflows/README.md
@@ -1,4 +1,4 @@
-Here you'll find a collection of workflows that demonstrate how to use Hamilton
+Here you'll find a collection of workflows that demonstrate how to use Apache Hamilton
in various LLM contexts.
For more examples that you can pull off-the-shelf, we invite you to checkout [hub.dagworks.io](https://hub.dagworks.io),
diff --git a/examples/LLM_Workflows/image_telephone/README.md b/examples/LLM_Workflows/image_telephone/README.md
index 8a117d64..bc88d297 100644
--- a/examples/LLM_Workflows/image_telephone/README.md
+++ b/examples/LLM_Workflows/image_telephone/README.md
@@ -5,10 +5,10 @@ of previously run images.
This example uses dataflows from the hub to do something fun with image captioning and generation.
Note that hamilton code is used rather than defined here. How the game is orchestrated
-is by using Hamilton's companion framework called [Burr](https://github.com/apache/burr).
+is by using Apache Hamilton's companion framework called [Burr](https://github.com/apache/burr).
-Burr is a simple way to define state machines and run them. It complements Hamilton by providing
-a way to save state, define loops and conditional branches. That is, it defines things that Hamilton
+Burr is a simple way to define state machines and run them. It complements Apache Hamilton by providing
+a way to save state, define loops and conditional branches. That is, it defines things that Apache Hamilton
cannot.
# Contents
diff --git a/examples/LLM_Workflows/knowledge_retrieval/README.md b/examples/LLM_Workflows/knowledge_retrieval/README.md
index 3ab3eb5b..6939307a 100644
--- a/examples/LLM_Workflows/knowledge_retrieval/README.md
+++ b/examples/LLM_Workflows/knowledge_retrieval/README.md
@@ -7,7 +7,7 @@ The point of the cookbook is to show how you can retrieve some knowledge, e.g. a
as input to OpenAI, using it in the context of conversation, where the OpenAI LLM can determine whether an external function should
be called to get more context.
-Hamilton is great for describing dataflows, i.e. DAGs. In this example, we will use Hamilton to define a DAG that
+Apache Hamilton is great for describing dataflows, i.e. DAGs. In this example, we will use Apache Hamilton to define a DAG that
encapsulates two functions that are called by OpenAI as housed in [functions.py](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/knowledge_retrieval/functions.py).
Specifically the logic for each is housed in two modules:
- getting articles from Arxiv, housed in [arxiv_articles.py](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/knowledge_retrieval/arxiv_articles.py)
@@ -25,7 +25,7 @@ Then to orchestrate this, state.py contains code to hold a conversation and func
to OpenAI and then exercise the two functions as appropriate.
Note: the original cookbook example is not production ready, and so to set expectations, in this example we get 80% of
-the way there translating it into Hamilton code. The remaining 20% is left as an exercise for the reader.
+the way there translating it into Apache Hamilton code. The remaining 20% is left as an exercise for the reader.
# Running the example
You just need to install the dependencies:
diff --git a/examples/LLM_Workflows/langchain_comparison/README.md b/examples/LLM_Workflows/langchain_comparison/README.md
index c3f5f846..971ec351 100644
--- a/examples/LLM_Workflows/langchain_comparison/README.md
+++ b/examples/LLM_Workflows/langchain_comparison/README.md
@@ -10,15 +10,15 @@ they have inside them.
Files prefixed with `lcel_` are LangChain examples.
Files prefixed with `vanilla_` are what you would write in vanilla Python.
-Files prefixed with `hamilton_` are the Hamilton equivalent of the examples.
+Files prefixed with `hamilton_` are the Apache Hamilton equivalent of the examples.
As you browse the files you'll see that:
1. LangChain's focus is on hiding details and making code terse.
-2. Hamilton's focus instead is on making code more readable, maintainable, and importantly customizeable.
+2. Apache Hamilton's focus instead is on making code more readable, maintainable, and importantly customizeable.
## Implications
-Don't be surprised that Hamilton's code is "longer" - that's by design. There is
-also little abstraction between you, and the underlying libraries with Hamilton.
+Don't be surprised that Apache Hamilton's code is "longer" - that's by design. There is
+also little abstraction between you, and the underlying libraries with Apache Hamilton.
With LangChain they're abstracted away, so you can't really see easily what's going on
underneath.
diff --git a/examples/LLM_Workflows/modular_llm_stack/README.md b/examples/LLM_Workflows/modular_llm_stack/README.md
index 3530ba33..7dcc005e 100644
--- a/examples/LLM_Workflows/modular_llm_stack/README.md
+++ b/examples/LLM_Workflows/modular_llm_stack/README.md
@@ -6,7 +6,7 @@ This example shows how to pull data from the HuggingFace datasets hub, create em
*DAG for OpenAI embeddings and Weaviate vector database*
-In addition, you'll see how Hamilton can help you create replaceable components. This flexibility, makes it easier to assess service providers and refactor code to fit your needs. The above and below DAGs were generated simply by changing a string value and a module import. Try to spot the differences!
+In addition, you'll see how Apache Hamilton can help you create replaceable components. This flexibility, makes it easier to assess service providers and refactor code to fit your needs. The above and below DAGs were generated simply by changing a string value and a module import. Try to spot the differences!
@@ -23,8 +23,8 @@ In addition, you'll see how Hamilton can help you create replaceable components.
# Example structure
- `run.py` contains the code to test the example. It uses `click` to provide a simple command interface.
- `data_module.py` contains the code to pull data from HuggingFace. The code is in a separate Python module since it doesn't depend on the other functionalities and could include more involved preprocessing.
-- `embedding_module.py` contains the code to embed text using either Cohere API, OpenAI API or SentenceTransformer library. The use of `@config.when` allows to have all options in the same Python module. This allows to quickly rerun your Hamilton DAG by simply changing your config. You'll see that functions share similar signature to enable interchangeability.
-- `lancedb_module.py`, `weaviate_module.py`, `marqo_module.py` and `pinecone_module.py` implement the same functionalities for each vector database. Having the same function names allows Hamilton to abstract away the implementation details and reinforce the notion that both modules shouldn't be loaded simultaneously.
+- `embedding_module.py` contains the code to embed text using either Cohere API, OpenAI API or SentenceTransformer library. The use of `@config.when` allows to have all options in the same Python module. This allows to quickly rerun your Apache Hamilton DAG by simply changing your config. You'll see that functions share similar signature to enable interchangeability.
+- `lancedb_module.py`, `weaviate_module.py`, `marqo_module.py` and `pinecone_module.py` implement the same functionalities for each vector database. Having the same function names allows Apache Hamilton to abstract away the implementation details and reinforce the notion that both modules shouldn't be loaded simultaneously.
- `docker-compose.yml` allows you to start a local instance of Weaviate ([More information](https://weaviate.io/developers/weaviate/installation/docker-compose)).
# How-to run the example
diff --git a/examples/LLM_Workflows/observability_openllmetry/README.md b/examples/LLM_Workflows/observability_openllmetry/README.md
index 9e5e72f4..d6e2141d 100644
--- a/examples/LLM_Workflows/observability_openllmetry/README.md
+++ b/examples/LLM_Workflows/observability_openllmetry/README.md
@@ -1,6 +1,6 @@
-# Monitor Hamilton with OpenTelemetry, OpenLLMetry and Traceloop
+# Monitor Apache Hamilton with OpenTelemetry, OpenLLMetry and Traceloop
-In this simple example, you'll learn how to use the `OpenTelemetryTracer` to emit traces of your Hamilton code using the OpenTelemetry format, in particular LLM applications.
+In this simple example, you'll learn how to use the `OpenTelemetryTracer` to emit traces of your Apache Hamilton code using the OpenTelemetry format, in particular LLM applications.
@@ -36,6 +36,6 @@ Having access to a [Traceloop account](https://www.traceloop.com/) and an API ke
For this example to work without Traceloop, you will need to set up your own [OpenTelemetry destination](https://opentelemetry.io/ecosystem/vendors/). We suggest using [Jaeger](https://www.jaegertracing.io/docs/1.47/getting-started/) and included Python code to route telemetry to it in `run.py`.
-## Should I still use the Hamilton UI?
+## Should I still use the Apache Hamilton UI?
-Absolutely! OpenTelemetry focsues on collecting telemetry about the internals of code and external API calls. It's a standard amongst web services. There's no conflict between the OpenTelemetry tracer and the tracking for the Hamilton UI. In fact, the Hamilton UI captures a superset of what OpenTelemetry allows, tailored to the Hamilton framework such as: visualizations, data lineage, summary statistics, and more utilities to improve your development experience. In the not too distant f [...]
+Absolutely! OpenTelemetry focsues on collecting telemetry about the internals of code and external API calls. It's a standard amongst web services. There's no conflict between the OpenTelemetry tracer and the tracking for the Apache Hamilton UI. In fact, the Apache Hamilton UI captures a superset of what OpenTelemetry allows, tailored to the Apache Hamilton framework such as: visualizations, data lineage, summary statistics, and more utilities to improve your development experience. In [...]
diff --git a/examples/LLM_Workflows/pdf_summarizer/README.md b/examples/LLM_Workflows/pdf_summarizer/README.md
index aee19a9b..e95f3a75 100644
--- a/examples/LLM_Workflows/pdf_summarizer/README.md
+++ b/examples/LLM_Workflows/pdf_summarizer/README.md
@@ -1,19 +1,19 @@
# (Yet another) LLM PDF Summarizer π
-Here's an extensible and production-ready PDF summarizer that you can run anywhere! The frontend uses streamlit, which communicates with a FastAPI backend powered by Hamilton. You give it a PDF file via the browser app and it returns you a text summary using the OpenAI API. If you want, you skip the browser inteface and directly access the `/summarize` endpoint with your document! Everything is containerized using Docker, so you should be able to run it where you please π.
+Here's an extensible and production-ready PDF summarizer that you can run anywhere! The frontend uses streamlit, which communicates with a FastAPI backend powered by Apache Hamilton. You give it a PDF file via the browser app and it returns you a text summary using the OpenAI API. If you want, you skip the browser inteface and directly access the `/summarize` endpoint with your document! Everything is containerized using Docker, so you should be able to run it where you please π.
## Why build this project?
-This project shows how easy it is to productionize Hamilton. Its function-centric declarative approach makes the code easy to read and extend. We invite you to clone the repo and customize to your needs! We are happy to help you via [Slack](https://hamilton-opensource.slack.com/join/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) and are excited to see what you build π
+This project shows how easy it is to productionize Apache Hamilton. Its function-centric declarative approach makes the code easy to read and extend. We invite you to clone the repo and customize to your needs! We are happy to help you via [Slack](https://hamilton-opensource.slack.com/join/shared_invite/zt-1bjs72asx-wcUTgH7q7QX1igiQ5bbdcg) and are excited to see what you build π
Here are a few ideas:
- Modify the streamlit `file_uploader` to allow sending batches of files through the UI
- Add PDF parsing and preprocessing to reduce number of tokens sent to OpenAI
-- Add Hamilton functions to gather metadata (file length, number of tokens, language, etc.) and return it via `SummaryResponse`
+- Add Apache Hamilton functions to gather metadata (file length, number of tokens, language, etc.) and return it via `SummaryResponse`
- Support other file formats; use the `@config.when()` decorator to add alternatives to the `raw_text()` function for PDFs
- Extract structured data from PDFs using open source models from the HuggingFace Hub.
-*The Hamilton execution DAG powering the backend*
+*The Apache Hamilton execution DAG powering the backend*
# Setup
diff --git a/examples/LLM_Workflows/pdf_summarizer/run_on_spark/README.md b/examples/LLM_Workflows/pdf_summarizer/run_on_spark/README.md
index 49794c56..24fdc088 100644
--- a/examples/LLM_Workflows/pdf_summarizer/run_on_spark/README.md
+++ b/examples/LLM_Workflows/pdf_summarizer/run_on_spark/README.md
@@ -1,6 +1,6 @@
# PDF Summarizer on Spark
-Here we show how you can run the same Hamilton dataflow, that we defined in the backend
+Here we show how you can run the same Apache Hamilton dataflow, that we defined in the backend
folder, on Spark (using two approaches). This is useful if you want to run the same dataflow
on a larger dataset, or have to run it on a cluster. Importantly this means you don't have
to rewrite your code, or have to change where/how you develop!
@@ -10,14 +10,14 @@ to rewrite your code, or have to change where/how you develop!
# File organization
- `summarization.py` this should be a carbon copy of the one in the backend folder.
- `run_with_columns.py` this contains the code to create a spark job and run the summarization code, but
-using the `@with_columns` syntax. This is a more ergonomic way to write a full spark job with Hamilton.
+using the `@with_columns` syntax. This is a more ergonomic way to write a full spark job with Apache Hamilton.
- `run.py` this contains the code to create a spark job and run the summarization code as UDFs.
# How `run_with_columns.py` works
We take the dataflow defined by `summarization.py` and execute it as a bunch
of row based UDFs on spark. The magic to do this happens in the `@with_columns` decorator.
-This approach allows you to put your entire pyspark workflow into Hamilton in an ergonomic way.
+This approach allows you to put your entire pyspark workflow into Apache Hamilton in an ergonomic way.
You can request any intermediate outputs of summarization.py as columns in the dataframe.
This is great for debugging and understanding your dataflow.
@@ -25,7 +25,7 @@ This is great for debugging and understanding your dataflow.
# How `run.py` works
We take the dataflow defined by `summarization.py` and execute it as a bunch
-of row based UDFs on spark. The magic to do this happens in the Hamilton PySparkUDFGraphAdapter.
+of row based UDFs on spark. The magic to do this happens in the Apache Hamilton PySparkUDFGraphAdapter.
The premise is that there is a central dataframe
that contains a column that maps to the required input. Which in this example
diff --git a/examples/LLM_Workflows/retrieval_augmented_generation/README.md b/examples/LLM_Workflows/retrieval_augmented_generation/README.md
index 99882b16..5a870120 100644
--- a/examples/LLM_Workflows/retrieval_augmented_generation/README.md
+++ b/examples/LLM_Workflows/retrieval_augmented_generation/README.md
@@ -1,7 +1,7 @@
-# Retrieval Augmented Generation app with Hamilton
+# Retrieval Augmented Generation app with Apache Hamilton
This application allows you to search arXiv for PDFs or import arbitrary PDF files and search over them using LLMs. For each file, the text is divided in chunks that are embedded with OpenAI and stored in Weaviate. When you query the system, the most relevant chunks are retrieved and a summary answer is generated using OpenAI.
-The ingestion and retrieval steps are implemented as dataflows with Hamilton and are exposed via FastAPI endpoints. The frontend is built with Streamlit and exposes the different functionalities via a simple web UI. Everything is packaged as containers with docker compose.
+The ingestion and retrieval steps are implemented as dataflows with Apache Hamilton and are exposed via FastAPI endpoints. The frontend is built with Streamlit and exposes the different functionalities via a simple web UI. Everything is packaged as containers with docker compose.
This example draws from previous simpler examples ([Knowledge Retrieval](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/knowledge_retrieval), [Modular LLM Stack](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/modular_llm_stack), [PDF Summarizer](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/pdf_summarizer)).
@@ -31,8 +31,8 @@ or you can do `docker compose logs -f` to tail the logs (ctrl+c to stop tailing
- [Docker Compose Networking](https://docs.docker.com/compose/networking/)
- [FastAPI /docs metadata](https://fastapi.tiangolo.com/tutorial/metadata/)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
-- [Hamilton Interactive online demo](https://www.tryhamilton.dev/)
-- [Hamilton Parallelizable/Collect](https://hamilton.dagworks.io/en/latest/concepts/customizing-execution/#dynamic-dags-parallel-execution)
+- [Apache Hamilton Interactive online demo](https://www.tryhamilton.dev/)
+- [Apache Hamilton Parallelizable/Collect](https://hamilton.dagworks.io/en/latest/concepts/customizing-execution/#dynamic-dags-parallel-execution)
- [OpenAI API docs](https://platform.openai.com/docs/introduction)
- [RAG sample architecture by Amazon](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-customize-rag.html)
- [Streamlit API documentation](https://docs.streamlit.io/library/api-reference)
diff --git a/examples/LLM_Workflows/scraping_and_chunking/README.md b/examples/LLM_Workflows/scraping_and_chunking/README.md
index 62000ac9..26dafd94 100644
--- a/examples/LLM_Workflows/scraping_and_chunking/README.md
+++ b/examples/LLM_Workflows/scraping_and_chunking/README.md
@@ -3,7 +3,7 @@ Scraping and chunking are an important part of any RAG dataflow. Typically they'
the start of your "backend" operations to populate for example your vector database.
## High Level Explanation
-Here we show how to model this process with Hamilton, but also we show how to avoid
+Here we show how to model this process with Apache Hamilton, but also we show how to avoid
dealing with executors and control flow logic that can make your code hard to maintain, test, and reuse.
For the latter case, see the example code below. You would typically see this in a scraping and chunking workflow to
parallelize it. `some_func` below would be some large function, or wrapper around logic to process each
@@ -29,7 +29,7 @@ def scrape(urls: list) -> list:
return all_data
```
##
-Instead, with Hamilton, you can write the processing logic INDEPENDENT of having to deal
+Instead, with Apache Hamilton, you can write the processing logic INDEPENDENT of having to deal
with the for loop and control logic to submit to the executor. This is a big win, because
it means you can easily unit test your code, reuse it, and then scale it to run in parallel without
coupling to a specific execution system.
@@ -65,7 +65,7 @@ def processed_article(article_text: str) -> list:
# do some processing, even saving it, etc.
return article_text
```
-Next we can then "parallelize" & "collect" it over inputs, i.e. "map" over it with various values. To tell Hamilton to
+Next we can then "parallelize" & "collect" it over inputs, i.e. "map" over it with various values. To tell Apache Hamilton to
do that we'd add the following functions to "sandwich" the code above:
```python
def url(urls_from_sitemap: list[str], max_urls: int = 1000) -> Parallelizable[str]:
@@ -77,7 +77,7 @@ def url(urls_from_sitemap: list[str], max_urls: int = 1000) -> Parallelizable[st
for url in urls_from_sitemap[0:max_urls]:
yield url
-# The previous Hamilton code could live here, or if in another module, Hamilton
+# The previous Apache Hamilton code could live here, or if in another module, Apache Hamilton
# would stitch the graph together correctly.
def collect_processed_articles(processed_article: Collect[list]) -> list:
@@ -87,7 +87,7 @@ def collect_processed_articles(processed_article: Collect[list]) -> list:
"""
return list(url_result)
```
-The magic is in the `Parallelizable` & `Collect` types. This tells Hamilton to run what is between them
+The magic is in the `Parallelizable` & `Collect` types. This tells Apache Hamilton to run what is between them
in parallel as a single task. For more information see the
[parallel documentation](https://hamilton.dagworks.io/en/latest/concepts/parallel-task/) and
[examples](https://github.com/apache/hamilton/tree/main/examples/parallelism).
@@ -98,7 +98,7 @@ Here is an image of the pipeline when run locally, or via ray or dask:
The pipeline is a simple one that:
-1. takes in a sitemap.xml file and creates a list of all the URLs in the file. Defaults to Hamilton's.
+1. takes in a sitemap.xml file and creates a list of all the URLs in the file. Defaults to Apache Hamilton's.
2. For each URL the process is then parallelized (green border).
3. each url is pulled and stripped to the relevant body of HTML.
4. the HTML is then chunked into smaller pieces -- returning langchain documents
@@ -116,11 +116,11 @@ Or you can open the notebook in google collab:<a target="_blank" href="https://c
### File Structure
Here we explain the file structure of the example:
- - `doc_pipeline.py` - the main file that contains the Hamilton code that defines the document chunking pipeline.
+ - `doc_pipeline.py` - the main file that contains the Apache Hamilton code that defines the document chunking pipeline.
- `run.py` - code that you would invoke to run `doc_pipeline` locally, or in a single python process.
- `run_dask.py` - code that you would invoke to run `doc_pipeline` on a Dask cluster / or dask locally.
- `run_ray.py` - code that you would invoke to run `doc_pipeline` on a Ray cluster / or ray locally.
- - `spark/doc_pipeline.py` - the main file that contains the Hamilton code that defines the document chunking pipeline,
+ - `spark/doc_pipeline.py` - the main file that contains the Apache Hamilton code that defines the document chunking pipeline,
but adjusted for PySpark.
- `spark/spark_pipeline.py` - code that you would invoke to run `spark/doc_pipeline` on a Spark cluster / or spark locally.
- `spark/README.md` - more details on running the Spark example and why the code differs slightly.
@@ -145,7 +145,7 @@ This example is a simple one, but it's easy to extend. For example, you could:
* you could also add a step to save the results to a database, or to a file system.
* you'd also likely tune the parallelism to ensure you don't DoS the resource you're hitting.
-## Hamilton over Langchain
-Hamilton is a general purpose tool, and what we've described here applies broadly
+## Apache Hamilton over Langchain
+Apache Hamilton is a general purpose tool, and what we've described here applies broadly
to any code that you might write: data, machine learning, LLMs, web processing, etc. You can
even use it with parts of LangChain!
diff --git a/examples/LLM_Workflows/scraping_and_chunking/spark/README.md b/examples/LLM_Workflows/scraping_and_chunking/spark/README.md
index 1ba66829..e67f0142 100644
--- a/examples/LLM_Workflows/scraping_and_chunking/spark/README.md
+++ b/examples/LLM_Workflows/scraping_and_chunking/spark/README.md
@@ -1,6 +1,6 @@
# PySpark version of the scraping and chunking example
-Here we show how you can integrate the Hamilton dataflow, that we defined previously
+Here we show how you can integrate the Apache Hamilton dataflow, that we defined previously
easily into a PySpark job. This is useful if you want to run the same dataflow on a larger dataset,
or have to run it on a cluster.
@@ -12,7 +12,7 @@ you to parallelize operations. This is useful if you have a large number of docu
# File organization
- `spark_pipeline.py` is the main dataflow for the spark job
- `doc_pipline.py` contains the code to parse and chunk the documents. It has a few adjustments to make
-it work with the `@with_columns` integration Hamilton has.
+it work with the `@with_columns` integration Apache Hamilton has.
# How to run
If you're on MacOS:
@@ -36,7 +36,7 @@ Here's why we need to make some minor adjustments to the code.
e.g. the LangChain chunker in this example, it cannot be used as a column. We have to convert it to a string
or a struct, or not include it in the "sub-DAG" that with_columns is creating.
2. The `@with_columns` decorator assumes that intermediate functions can be saved as columns to the dataframe.
-That's how it strings together computation. If this is not possible, you have to tell Hamilton to not include
+That's how it strings together computation. If this is not possible, you have to tell Apache Hamilton to not include
the function in the "sub-DAG" that with_columns is creating.
## Handling langchain's Document type
@@ -48,7 +48,7 @@ In `doc_pipline.py` you'll see we define a variable that is a list of the functi
"sub-DAG" that with_columns is creating. This is a quick way to label or specify what transforms,
can be made into columns. `@with_columns` will then reference this list of functions to know what to include.
That way `html_chunker` and `text_chunker` are not included in the "sub-DAG" that `@with_columns` is creating, and
-are then run once and bound to the UDFs the Hamilton is creating underneath.
+are then run once and bound to the UDFs the Apache Hamilton is creating underneath.
# Caveats to think about in real life
## Partitions & number of executors
diff --git a/examples/README.md b/examples/README.md
index 4f0e5f68..0fba99e6 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -3,7 +3,7 @@
Here you'll find various examples, some relatively simple, to a few that are more complex. As you can see
there are MANY examples, that are mostly organized by a thematic topic that matches the folder name they're in.
-Note: [hub.dagworks.io](https://hub.dagworks.io/) is also a good spot to find Hamilton examples.
+Note: [hub.dagworks.io](https://hub.dagworks.io/) is also a good spot to find Apache Hamilton examples.
If you have questions, or need help with these examples,
join us on [slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g), and we'll try to help!
diff --git a/examples/airflow/README.md b/examples/airflow/README.md
index de1dada8..603b544a 100644
--- a/examples/airflow/README.md
+++ b/examples/airflow/README.md
@@ -1,22 +1,22 @@
-# Hamilton and Airflow
+# Apache Hamilton and Airflow
-In this example, we're going to show how to run Hamilton within a Airflow task. Both tools are used to author DAGs with Python code, but they operate on different levels:
+In this example, we're going to show how to run Apache Hamilton within a Airflow task. Both tools are used to author DAGs with Python code, but they operate on different levels:
- **Airflow** is an orchestrator written in Python. Its purpose is to launch tasks (which can be of any kind; Python, SQL, Bash, etc.) and make sure they complete.
-- **Hamilton** is a data transformation framework. It helps developer write Python code that is modular and reusable, and that can be executed as a DAG.
+- **Apache Hamilton** is a data transformation framework. It helps developer write Python code that is modular and reusable, and that can be executed as a DAG.
-Therefore, Hamilton allows you to write modular and easier to maintain code that can be used within or outside Airflow. This results in simpler Airflow DAG that will improve maintainability and might improve performance.
+Therefore, Apache Hamilton allows you to write modular and easier to maintain code that can be used within or outside Airflow. This results in simpler Airflow DAG that will improve maintainability and might improve performance.
## File organization
-- `/dags/hamilton/hamilton_how_to_dag.py` shows the basics of Hamilton and illustrates how to integrate with Airflow.
+- `/dags/hamilton/hamilton_how_to_dag.py` shows the basics of Apache Hamilton and illustrates how to integrate with Airflow.
- `/dags/hamilton/absenteeism_prediction_dag.py` shows a concrete example of loading data, training a machine learning model, and evaluating it.
-- Each example is powered by Python modules under `/plugins/function_modules` and `/plugins/absenteeism` respectively (details about this in the How-to DAG). Under `/docs`, you'll find the DAG visualization of these Hamilton modules.
+- Each example is powered by Python modules under `/plugins/function_modules` and `/plugins/absenteeism` respectively (details about this in the How-to DAG). Under `/docs`, you'll find the DAG visualization of these Apache Hamilton modules.
- For the purpose of this example repository, data will be read from and written to `/plugins/data` since it is easily accessible to Airflow. This shouldn't be used in production settings.
## Airflow setup
We will use custom Docker containers based on the [official Airflow `docker-compose.yaml`](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html).
-0. git clone the Hamilton repository
+0. git clone the Apache Hamilton repository
1. From the terminal, go to the airflow example directory `hamilton/examples/airflow/`
2. Create a `.env` file with your Airflow UID using: `echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env`
3. Create the Docker containers using: `docker compose up --build` This can take a few minutes. Afterwards, your Airflow Docker containers should be running.
@@ -29,9 +29,9 @@ We will use custom Docker containers based on the [official Airflow `docker-comp
The DAG that has simple linear structure A -> B -> C will experience less delays in task scheduling than DAG that has a deeply nested tree structure with exponentially growing number of depending tasks for example. If you can make your DAGs more linear - where at single point in execution there are as few potential candidates to run among the tasks, this will likely improve overall scheduling performance.
-3. **Separate infrastructure from data science concerns**. Airflow is responsible for orchestrating tasks and is typically owned handled by data engineers. Hamilton is responsible for authoring clean, reusable, and maintainable data transformations and is typically owned by data scientists. By integrating both tools, these two personas/teams have clearer ownership and version control over their codebases, and it can promote reusability of both Airflow and Hamilton code. Notably, one can [...]
+3. **Separate infrastructure from data science concerns**. Airflow is responsible for orchestrating tasks and is typically owned handled by data engineers. Apache Hamilton is responsible for authoring clean, reusable, and maintainable data transformations and is typically owned by data scientists. By integrating both tools, these two personas/teams have clearer ownership and version control over their codebases, and it can promote reusability of both Airflow and Apache Hamilton code. Not [...]
-4. **Write efficient Python code efficiently**. With Python being the 3rd most popular programming language in 2023, most data professionals should be able to pick up Airflow and Hamilton quickly. However, production systems are composed of multiple services (database, datawarehouse, compute cluster, cache, serverless functions, etc.) most of which are SQL-based or have a Python SDK. This puts strains on engineers that need to learn and maintain this sprawling codebase. For orchestration [...]
+4. **Write efficient Python code efficiently**. With Python being the 3rd most popular programming language in 2023, most data professionals should be able to pick up Airflow and Apache Hamilton quickly. However, production systems are composed of multiple services (database, datawarehouse, compute cluster, cache, serverless functions, etc.) most of which are SQL-based or have a Python SDK. This puts strains on engineers that need to learn and maintain this sprawling codebase. For orches [...]
# Notes and limitations
diff --git a/examples/async/README.md b/examples/async/README.md
index 8ef0fb49..0349599c 100644
--- a/examples/async/README.md
+++ b/examples/async/README.md
@@ -1,4 +1,4 @@
-# Hamilton + Async
+# Apache Hamilton + Async
This is currently an experimental feature, allowing one to run a hamilton DAG composed (entirely or partially) of async functions.
diff --git a/examples/aws/glue/README.md b/examples/aws/glue/README.md
index e6e70912..65248395 100644
--- a/examples/aws/glue/README.md
+++ b/examples/aws/glue/README.md
@@ -1,6 +1,6 @@
-# Deploy Hamilton Functions as an AWS Glue Job
+# Deploy Apache Hamilton Functions as an AWS Glue Job
-[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html) using Hamilton functions on AWS Glue.
+[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html) using Apache Hamilton functions on AWS Glue.
## Prerequisites
@@ -8,9 +8,9 @@
## Step-by-Step Guide
-### 1. Build wheel with Hamilton functions
+### 1. Build wheel with Apache Hamilton functions
-First things first, AWS Glue jobs run a single python script, but you can include external code (like our Hamilton functions) by adding it as a python wheel. So, let's package our code and get it ready for action.
+First things first, AWS Glue jobs run a single python script, but you can include external code (like our Apache Hamilton functions) by adding it as a python wheel. So, let's package our code and get it ready for action.
- **Install build package:**
diff --git a/examples/aws/lambda/README.md b/examples/aws/lambda/README.md
index 7d407a64..3f929cc4 100644
--- a/examples/aws/lambda/README.md
+++ b/examples/aws/lambda/README.md
@@ -1,8 +1,8 @@
-# Deploy Hamilton in AWS Lambda
+# Deploy Apache Hamilton in AWS Lambda
[AWS Lambda](https://aws.amazon.com/lambda/) - serverless computation service in AWS.
-Here we have an example how to deploy "hello-world" AWS Lambda with Hamilton functions.
+Here we have an example how to deploy "hello-world" AWS Lambda with Apache Hamilton functions.
This example is based on the official instruction: https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-instructions
## Prerequisites
diff --git a/examples/aws/sagemaker/README.md b/examples/aws/sagemaker/README.md
index 20c5507f..e7c8f3b9 100644
--- a/examples/aws/sagemaker/README.md
+++ b/examples/aws/sagemaker/README.md
@@ -1,6 +1,6 @@
-# Run Hamilton Functions as an AWS SageMaker Processing Job
+# Run Apache Hamilton Functions as an AWS SageMaker Processing Job
-[AWS SageMaker](https://aws.amazon.com/sagemaker/) is a comprehensive platform that facilitates the creation, training, and deployment of machine learning (ML) models. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html) using Hamilton functions on SageMaker.
+[AWS SageMaker](https://aws.amazon.com/sagemaker/) is a comprehensive platform that facilitates the creation, training, and deployment of machine learning (ML) models. This guide demonstrates deploying a "hello-world" [processing job](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html) using Apache Hamilton functions on SageMaker.
## Prerequisites
diff --git a/examples/caching/README.md b/examples/caching/README.md
index 82a038bf..ac4732a3 100644
--- a/examples/caching/README.md
+++ b/examples/caching/README.md
@@ -1,6 +1,6 @@
# Caching
-This directory contains tutorial notebooks for the Hamilton caching feature.
+This directory contains tutorial notebooks for the Apache Hamilton caching feature.
- `tutorial.ipynb`: the main tutorial for caching
- `in_memory_tutorial.ipynb`: How to use caching without writing metadata and results to file.
diff --git a/examples/caching_nodes/README.md b/examples/caching_nodes/README.md
index bdc1bb88..5c3343de 100644
--- a/examples/caching_nodes/README.md
+++ b/examples/caching_nodes/README.md
@@ -1,8 +1,8 @@
# Caching
-There are four general approaches to caching in Hamilton.
+There are four general approaches to caching in Apache Hamilton.
-1. You do it yourself outside of Hamilton and use the [`overrides`](https://hamilton.dagworks.io/en/latest/reference/drivers/#short-circuiting-some-dag-computation) argument in `.execute/.materialize(..., overrides={...})` to inject pre-computed values into the graph. That is, you run your code, save the things you want, and then you load them and inject them using `overrides=`. TODO: show example.
+1. You do it yourself outside of Apache Hamilton and use the [`overrides`](https://hamilton.dagworks.io/en/latest/reference/drivers/#short-circuiting-some-dag-computation) argument in `.execute/.materialize(..., overrides={...})` to inject pre-computed values into the graph. That is, you run your code, save the things you want, and then you load them and inject them using `overrides=`. TODO: show example.
2. You use the data savers & data loaders. This is similar to the above, but instead you use the [Data Savers & Data Loaders (i.e. materializers)](https://hamilton.dagworks.io/en/latest/reference/io/available-data-adapters/) to save & then load and inject data in. TODO: show example.
3. You use the `CachingGraphAdapter`, which requires you to tag functions to cache along with the serialization format.
4. You use the `DiskCacheAdapter`, which uses the `diskcache` library to store the results on disk.
diff --git a/examples/caching_nodes/diskcache_adapter/README.md b/examples/caching_nodes/diskcache_adapter/README.md
index 80025ee3..a4f3ed3e 100644
--- a/examples/caching_nodes/diskcache_adapter/README.md
+++ b/examples/caching_nodes/diskcache_adapter/README.md
@@ -15,7 +15,7 @@ Disk cache has great features to:
> β The default `Disk` serializes objects using the `pickle` module. Changing Python or library versions could break your
> cache (both keys and values). Learn more about [caveats](https://grantjenks.com/docs/diskcache/tutorial.html#caveats).
-> β To store artifacts robustly, please use Hamilton materializers or the
+> β To store artifacts robustly, please use Apache Hamilton materializers or the
> [CachingGraphAdapter](https://github.com/apache/hamilton/tree/main/examples/caching_nodes/caching_graph_adatper) instead.
> The `CachingGraphAdapter` stores tagged nodes directly on the file system using common formats (JSON, CSV, Parquet, etc.).
> However, it isn't aware of your function version and requires you to manually manage your disk space.
diff --git a/examples/cli/DOCS.md b/examples/cli/DOCS.md
index 9ac0a515..ab9f015a 100644
--- a/examples/cli/DOCS.md
+++ b/examples/cli/DOCS.md
@@ -1,6 +1,6 @@
# `hamilton`
-Hamilton CLI
+Apache Hamilton CLI
**Usage**:
diff --git a/examples/cli/README.md b/examples/cli/README.md
index 346d905d..04b6cd2f 100644
--- a/examples/cli/README.md
+++ b/examples/cli/README.md
@@ -1,6 +1,6 @@
-# Hamilton CLI
+# Apache Hamilton CLI
-The Hamilton CLI allows to build `Driver` objects from the command line.
+The Apache Hamilton CLI allows to build `Driver` objects from the command line.
# Installation
Install dependencies with (it only needs `typer`)
@@ -14,7 +14,7 @@ Test the installation with
# Features
## Commands
-- `build`: creates a Hamilton `Driver` from specified modules. It"s useful to validate the dataflow definition
+- `build`: creates a Apache Hamilton `Driver` from specified modules. It"s useful to validate the dataflow definition
- `validate`: calls `Driver.validate_execution()` for a set of `inputs` and `overrides` passed through the `--context` option.
- `view`: calls `dr.display_all_functions()` on the built `Driver`
- `version`: generates node hashes based on their source code, and a dataflow hash from the collection of node hashes.
diff --git a/examples/contrib/README.md b/examples/contrib/README.md
index 2db9910b..b0263421 100644
--- a/examples/contrib/README.md
+++ b/examples/contrib/README.md
@@ -1,6 +1,6 @@
# Import Community Dataflows
-In this example, we show you how to import and use dataflows from the [Hamilton Dataflow Hub](https://hub.dagworks.io/docs/).
+In this example, we show you how to import and use dataflows from the [Apache Hamilton Dataflow Hub](https://hub.dagworks.io/docs/).
You can use them either directly or pull and edit a local copy.
# Setup
diff --git a/examples/dagster/README.md b/examples/dagster/README.md
index aaae37c6..b6534ad3 100644
--- a/examples/dagster/README.md
+++ b/examples/dagster/README.md
@@ -1,14 +1,14 @@
-# Dagster + Hamilton
+# Dagster + Apache Hamilton
-This repository compares how to build dataflows with macro orchestrator Dagster and the micro orchestrator Hamilton.
+This repository compares how to build dataflows with macro orchestrator Dagster and the micro orchestrator Apache Hamilton.
-> see the [side-by-side comparison](https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/) in the Hamilton documentation
+> see the [side-by-side comparison](https://hamilton.dagworks.io/en/latest/code-comparisons/dagster/) in the Apache Hamilton documentation
## Content
- `dagster_code/` includes code from the [Dagster tutorial](https://docs.dagster.io/tutorial) to load data and compute statistics from the website [HackerNews](https://news.ycombinator.com/).
-- `hamilton_code/` is a refactor of `dagster_tutorial/` using the Hamilton framework.
+- `hamilton_code/` is a refactor of `dagster_tutorial/` using the Apache Hamilton framework.
-Each directory contains instructions on how to run the code. We suggest going through the Dagster code first, then read the Hamilton refactor.
+Each directory contains instructions on how to run the code. We suggest going through the Dagster code first, then read the Apache Hamilton refactor.
## Setup
1. Create a virtual environment and activate it
@@ -16,7 +16,7 @@ Each directory contains instructions on how to run the code. We suggest going th
python -m venv venv && . venv/bin/active
```
-2. Install requirements for both Dagster and Hamilton
+2. Install requirements for both Dagster and Apache Hamilton
```console
pip install -r requirements.txt
```
diff --git a/examples/dagster/hamilton_code/README.md b/examples/dagster/hamilton_code/README.md
index a2b6bf90..3bc4b26c 100644
--- a/examples/dagster/hamilton_code/README.md
+++ b/examples/dagster/hamilton_code/README.md
@@ -1,16 +1,16 @@
-# Hamilton
+# Apache Hamilton
-> Hamilton dataflow materialization plan
+> Apache Hamilton dataflow materialization plan
## File structure
-The Hamilton refactor achieves the same data transformations. The project fits in essentially 2 files:
+The Apache Hamilton refactor achieves the same data transformations. The project fits in essentially 2 files:
- `dataflow.py` contains regular Python functions to define the dataflow (equivalent to Dagster `tutorial/assets.py`).
- `run.py` contains the "driver code" to load and execute the dataflow (equivalent to Dagster `tutorial/__init__.py`).
- `notebook.ipynb` contains code to execute the dataflow (similar code to `run.py`), but allows for an interactive exploration of results.
-The directory also include `mock_api.py` which simulates a `ressource` in the Dagster tutorial, and `dataflow.png` which is an artifact generated by the Hamilton `Driver`.
+The directory also include `mock_api.py` which simulates a `ressource` in the Dagster tutorial, and `dataflow.png` which is an artifact generated by the Apache Hamilton `Driver`.
## Instructions
1. Run the code
@@ -19,6 +19,6 @@ The directory also include `mock_api.py` which simulates a `ressource` in the Da
```
## Going further
-- Learn the basics of Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
+- Learn the basics of Apache Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
- Visit [tryhamilton.dev](tryhamilton.dev) for an interactive tutorial in your browser
- Visit the [DAGWorks blog](https://blog.dagworks.io/) for more detailed guides
diff --git a/examples/dask/README.md b/examples/dask/README.md
index 40cf717c..13760bb5 100644
--- a/examples/dask/README.md
+++ b/examples/dask/README.md
@@ -1,8 +1,8 @@
-# Scaling Hamilton on Dask
+# Scaling Apache Hamilton on Dask
[Dask](https://dask.org) is a general purpose framework that allows for parallel
computation on a local machine, as well as scaling to a
Dask cluster.
See the `hello_world` example for how you might accomplish
-scaling Hamilton on Dask.
+scaling Apache Hamilton on Dask.
diff --git a/examples/dask/community_demo/README.md b/examples/dask/community_demo/README.md
index fc9fb194..d674e98b 100644
--- a/examples/dask/community_demo/README.md
+++ b/examples/dask/community_demo/README.md
@@ -2,14 +2,14 @@
This example was demo'ed at the Dask community day for July 2023.
-In this example we showcase using the [Hamilton DaskGraphAdapter](https://hamilton.dagworks.io/en/latest/reference/graph-adapters/DaskGraphAdapter/#h-dask-daskgraphadapter)
-to inject dask.delayed() as it's walking the DAG that is defined with Hamilton.
+In this example we showcase using the [Apache Hamilton DaskGraphAdapter](https://hamilton.dagworks.io/en/latest/reference/graph-adapters/DaskGraphAdapter/#h-dask-daskgraphadapter)
+to inject dask.delayed() as it's walking the DAG that is defined with Apache Hamilton.
File organization:
* `my_functions.py` houses the logic that we want to compute. Note how the functions are named, and what input
parameters they require. That is how we create a DAG modeling the dataflow we want to happen.
-* `demo_day_notebook.ipynb` is the notebook that was shown in the demo. At the top it shows using Hamilton, and running
+* `demo_day_notebook.ipynb` is the notebook that was shown in the demo. At the top it shows using Apache Hamilton, and running
the code without delegating to Dask, and then at the bottom it shows how to use the DaskGraphAdapter that augments
how the DAG is walked, because it calls dask.delayed on every function.
@@ -34,5 +34,5 @@ This is a simple DAG that we construct here. But for reference it looks like thi
# References:
* [Dask](https://dask.org/)
-* [Hamilton Docs](https://hamilton.dagworks.io/en/latest/)
-* [Try Hamilton in your browser](https://www.tryhamilton.dev/)
+* [Apache Hamilton Docs](https://hamilton.dagworks.io/en/latest/)
+* [Try Apache Hamilton in your browser](https://www.tryhamilton.dev/)
diff --git a/examples/dask/hello_world/README.md b/examples/dask/hello_world/README.md
index b39c66d4..61c58463 100644
--- a/examples/dask/hello_world/README.md
+++ b/examples/dask/hello_world/README.md
@@ -1,7 +1,7 @@
-# Hamilton on Dask
+# Apache Hamilton on Dask
Here we have a hello world example showing how you can
-take some Hamilton functions and then easily run them
+take some Apache Hamilton functions and then easily run them
in a distributed setting via dask.
Note: please read this [dask best practices post](https://docs.dask.org/en/stable/dataframe-best-practices.html);
diff --git a/examples/data_loaders/README.md b/examples/data_loaders/README.md
index e5940c2b..9d07b325 100644
--- a/examples/data_loaders/README.md
+++ b/examples/data_loaders/README.md
@@ -1,8 +1,8 @@
# Data Loaders
-Among multiple uses, Hamilton excels at building maintainable, scalable representations of ETLs.
+Among multiple uses, Apache Hamilton excels at building maintainable, scalable representations of ETLs.
If you've read through the other guides, it should be pretty clear how hamilton enables transformations (the T in ETL/ELT).
-In this example, we'll talk about an approach to _Extracting_ data, and how Hamilton enables you to build out extracts,
+In this example, we'll talk about an approach to _Extracting_ data, and how Apache Hamilton enables you to build out extracts,
in a scalable, pluggable way. For example, being able to switch where data is loaded between development and production
is useful, since you might only want a subsample in development, or even load it from a different source.
Here we'll show you how you can achieve this without cluttering your code with `if else`,
diff --git a/examples/data_quality/pandera/README.md b/examples/data_quality/pandera/README.md
index 42d2cd32..f9679b50 100644
--- a/examples/data_quality/pandera/README.md
+++ b/examples/data_quality/pandera/README.md
@@ -1,5 +1,5 @@
# Pandera Data Quality Example
-Here we show how one can define some transformation logic that uses the new Hamilton data quality feature with [Pandera](https://pandera.readthedocs.io/)
+Here we show how one can define some transformation logic that uses the new Apache Hamilton data quality feature with [Pandera](https://pandera.readthedocs.io/)
In addition, we also show how you can manage execution on top of Ray, Pandas on Spark.
This example is ALMOST IDENTICAL to `examples/data_quality/simple` except that it instead uses Pandera as input to
@@ -30,7 +30,7 @@ this is where we would make that happen.
should happily run on top of Dask, and Ray.
* feature_logic_spark.py - this module contains some feature transformation code specific to running on top of Pandas on Spark.
Specifically, note that the data types checked against are different than in `feature_logic.py`.
-* run.py - this is the default Hamilton way of materializing features.
+* run.py - this is the default Apache Hamilton way of materializing features.
* run_dask.py - this shows how one would materialize features using Dask.
* run_ray.py - this shows how one would materialize features using Ray.
* run_spark.py - this shows how one would materialize features using Pandas on Spark.
@@ -48,22 +48,22 @@ Note: `importance` is not specified in the `@check_output`decorators found in fe
behavior is therefore invoked, which is to log a "warning" and not stop execution. If stopping execution is desired,
`importance="fail"` should then be added to the decorators; more centralized control is going to be added in future releases.
-### Normal Hamilton
+### Normal Apache Hamilton
> pip install -r requirements.txt
> python run.py
-### Hamilton on Dask
+### Apache Hamilton on Dask
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-dask.txt
> python run_dask.py
-### Hamilton on Ray
+### Apache Hamilton on Ray
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-ray.txt
> python run_ray.py
-### Hamilton on Pandas on Spark
+### Apache Hamilton on Pandas on Spark
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-spark.txt
> python run_spark.py
@@ -73,7 +73,7 @@ It is best practice to create a python virtual environment for each project/exam
Again you'll see that the visualizations don't change much between the different ways of executing. But to help you
visualize what's going on, here is the output of `visualize_execution` for each of them.
-## Vanilla Hamilton
+## Vanilla Apache Hamilton
## Dask
diff --git a/examples/data_quality/simple/README.md b/examples/data_quality/simple/README.md
index 11706b3e..3bc7cadc 100644
--- a/examples/data_quality/simple/README.md
+++ b/examples/data_quality/simple/README.md
@@ -28,7 +28,7 @@ here as an exercise for the reader.
this is where we would make that happen.
* feature_logic.py - this module contains some feature transformation code. It is annotated with `@check_output` and also
shows how one might support running the same code on Spark, Dask, and Ray, if you need to change how some code works.
-* run.py - this is the default Hamilton way of materializing features.
+* run.py - this is the default Apache Hamilton way of materializing features.
* run_dask.py - this shows how one would materialize features using Dask.
* run_ray.py - this shows how one would materialize features using Ray.
* run_spark.py - this shows how one would materialize features using Pandas on Spark.
@@ -46,22 +46,22 @@ Note: `importance` is not specified in the `@check_output`decorators found in fe
behavior is therefore invoked, which is to log a "warning" and not stop execution. If stopping execution is desired,
`importance="fail"` should then be added to the decorators; more centralized control is going to be added in future releases.
-### Normal Hamilton
+### Normal Apache Hamilton
> pip install -r requirements.txt
> python run.py
-### Hamilton on Dask
+### Apache Hamilton on Dask
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-dask.txt
> python run_dask.py
-### Hamilton on Ray
+### Apache Hamilton on Ray
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-ray.txt
> python run_ray.py
-### Hamilton on Pandas on Spark
+### Apache Hamilton on Pandas on Spark
It is best practice to create a python virtual environment for each project/example. We omit showing that step here.
> pip install -r requirements-spark.txt
> python run_spark.py
@@ -71,7 +71,7 @@ It is best practice to create a python virtual environment for each project/exam
Again you'll see that the visualizations don't change much between the different ways of executing. But to help you
visualize what's going on, here is the output of `visualize_execution` for each of them.
-## Vanilla Hamilton
+## Vanilla Apache Hamilton
## Dask
diff --git a/examples/dbt/README.md b/examples/dbt/README.md
index 12b9116b..18df5b37 100644
--- a/examples/dbt/README.md
+++ b/examples/dbt/README.md
@@ -1,20 +1,20 @@
-# Hamilton and DBT
+# Apache Hamilton and DBT
-In this example, we're going to show you how easy it is to run Hamilton inside a dbt task. Making use of DBT's exciting new
+In this example, we're going to show you how easy it is to run Apache Hamilton inside a dbt task. Making use of DBT's exciting new
[python API](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models), we can blend the two frameworks seamlessly.
-While the two frameworks might look similar at first glance, DBT and Hamilton are actually quite complementary.
+While the two frameworks might look similar at first glance, DBT and Apache Hamilton are actually quite complementary.
-- DBT is best at managing SQL logic and handling materialization, while Hamilton excels at modeling transforms in python
-- DBT contains its own orchestration capabilities, whereas Hamilton often relies on an external framework to run the code
-- DBT does not model micro-level transformations, whereas Hamilton thrives at enabling a user to specify them in a readable, maintainable way.
-- DBT is focused on analytic/warehouse-level transformations, whereas Hamilton can thrive at expressing ML-specific transforms.
+- DBT is best at managing SQL logic and handling materialization, while Apache Hamilton excels at modeling transforms in python
+- DBT contains its own orchestration capabilities, whereas Apache Hamilton often relies on an external framework to run the code
+- DBT does not model micro-level transformations, whereas Apache Hamilton thrives at enabling a user to specify them in a readable, maintainable way.
+- DBT is focused on analytic/warehouse-level transformations, whereas Apache Hamilton can thrive at expressing ML-specific transforms.
At a high-level, DBT can help you get the data/run large-scale operations in your warehouse,
-while Hamilton can help you make a model out of it.
+while Apache Hamilton can help you make a model out of it.
To demonstrate this, we've taken one of our favorite examples of writing data science code [xLaszlo's code quality for DS tutorial](https://github.com/xLaszlo/CQ4DS-notebook-sklearn-refactoring-exercise),
-and re-written it using a combination of DBT + Hamilton. This models the classic titanic problem.
+and re-written it using a combination of DBT + Apache Hamilton. This models the classic titanic problem.
In this case we're using [FAL](https://github.com/fal-ai/fal) to help run python in dbt -- it enables us to manage environments,
import packages happily, etc...
@@ -70,19 +70,19 @@ We've organized the code into two separate DBT models:
uglier python problems we hit along the way.
## Visualizing Execution
-Here is the DAG generated by Hamilton for the above example:
+Here is the DAG generated by Apache Hamilton for the above example:
# Future Directions
-This is just a start, and we think that Hamilton + DBT have a long/exciting future together. In particular, we could:
+This is just a start, and we think that Apache Hamilton + DBT have a long/exciting future together. In particular, we could:
-1. Compile Hamilton to DBT for orchestration -- the new [SQL adapter](https://github.com/apache/hamilton/issues/197) we're working on would compile nicely to a dbt task.
+1. Compile Apache Hamilton to DBT for orchestration -- the new [SQL adapter](https://github.com/apache/hamilton/issues/197) we're working on would compile nicely to a dbt task.
2. Add more natural integration -- including a dbt plugin for a hamilton task.
3. Add more examples with different SQL dialects/different python dialects. _hint_: _we're looking for contributors..._
If you're excited by any of this, drop on by! Some resources to get you help:
-- [Hamilton slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g)
+- [Apache Hamilton slack channel](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g)
- [DBT support](https://docs.getdbt.com/docs/dbt-support)
- [xLaszlo's CQ4DS discord](https://discord.gg/8uUZNMCad2)
diff --git a/examples/dlt/README.md b/examples/dlt/README.md
index 124487b0..1fb36171 100644
--- a/examples/dlt/README.md
+++ b/examples/dlt/README.md
@@ -1,14 +1,14 @@
# dlt
-This example shows dlt + Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this [documentation page](https://hamilton.dagworks.io/en/latest/integrations/dlt/).
+This example shows dlt + Apache Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this [documentation page](https://hamilton.dagworks.io/en/latest/integrations/dlt/).
# Content
It includes a pipeline to ingest messages from Slack channels and generate threads summaries.
- `slack/` is the code imported from dlt for the Slack `Source`
- `.dlt/` contains the source's version, config, and secrets for the dlt `Pipeline`
-- `transform.py` contains the Hamilton code to transform data and build the `threads` table
-- `run.py` contains the code to execution the dlt `Pipeline` and the Hamilton dataflow.
+- `transform.py` contains the Apache Hamilton code to transform data and build the `threads` table
+- `run.py` contains the code to execution the dlt `Pipeline` and the Apache Hamilton dataflow.
- `notebook.ipynb` contains the equivalent of both `transform.py` and `run.py` allowing you to explore the code interactively.
# Set up
diff --git a/examples/due_date_probabilities/README.md b/examples/due_date_probabilities/README.md
index 9c69da68..1c0a1d12 100644
--- a/examples/due_date_probabilities/README.md
+++ b/examples/due_date_probabilities/README.md
@@ -1,6 +1,6 @@
-# Modeling Pregnancy Due Dates using Hamilton
+# Modeling Pregnancy Due Dates using Apache Hamilton
-This is an example of developing and applying a simple statistical model using Hamilton
+This is an example of developing and applying a simple statistical model using Apache Hamilton
for a very common problem. What is the probability that a baby will be born (before, on, after)
X date?
@@ -15,8 +15,8 @@ You can read the full description in the [post](https://blog.dagworks.io/p/181bb
In this, you'll find the following files:
1. [notebook.ipynb](notebook.ipynb) - The Jupyter notebook that walks through the process of developing/running the model.
-2. [base_dates.py] -- a Hamilton module that generates date-related series
-3. [probability_estimation.py] -- a Hamilton module that computes a statistical model (estimating parameters for probability due dates)
-4. [probabilities.py] -- a Hamilton module that runs that model over a set of dates
+2. [base_dates.py] -- a Apache Hamilton module that generates date-related series
+3. [probability_estimation.py] -- a Apache Hamilton module that computes a statistical model (estimating parameters for probability due dates)
+4. [probabilities.py] -- a Apache Hamilton module that runs that model over a set of dates
Run the notebook to see the process of developing the model and running it.
diff --git a/examples/experiment_management/README.md b/examples/experiment_management/README.md
index 549aaa21..355165d1 100644
--- a/examples/experiment_management/README.md
+++ b/examples/experiment_management/README.md
@@ -1,6 +1,6 @@
-# Hamilton Experiment Manager
+# Apache Hamilton Experiment Manager
-Add a hook to your Hamilton Driver to log runs and visualize artifacts and metadata! The server is built using FastAPI + FastUI allowing to easily integrate the server within your app or extend the UI.
+Add a hook to your Apache Hamilton Driver to log runs and visualize artifacts and metadata! The server is built using FastAPI + FastUI allowing to easily integrate the server within your app or extend the UI.
<p align="center">
<img src="./showcase.gif" height=600, width=auto/>
@@ -15,12 +15,12 @@ Add a hook to your Hamilton Driver to log runs and visualize artifacts and metad
Use `pip install sf-hamilton[experiments]` to install both the hook and the server with their dependencies
## How to use the ExperimentTracker hook
-The `ExperimentTracker` hook can be added to your Hamilton Driver definition to automatically log metadata about the run and materialized results.
+The `ExperimentTracker` hook can be added to your Apache Hamilton Driver definition to automatically log metadata about the run and materialized results.
1. Create the `ExperimentTracker` hook object:
- `experiment_name`: name to organize related runs. Is used to create directories and displayed in the UI
- `base_directory`: path where the metadata cache and subdirectories to store artifacts will be created. Default is `./experiments`.
-2. Create the Hamilton Driver and pass the `ExperimentTracker` to `with_adapters()` method
+2. Create the Apache Hamilton Driver and pass the `ExperimentTracker` to `with_adapters()` method
3. Define materializers for each artifacts you care about. The UI provides rich support for `parquet`, `csv`, and `json`.
- β Make sure to use relative paths (ones that don't start with `/`) for artifacts to be stored in run directories.
4. Call `dr.materialize()` to launch run for which metadata and artifacts will be tracked.
@@ -44,7 +44,7 @@ Then navigate to `http://localhost:8123` to view the experiment manager! Then yo
from hamilton import driver
from hamilton.plugins import h_experiments
-import my_functions # <- your Hamilton module
+import my_functions # <- your Apache Hamilton module
# 1. create the hook
diff --git a/examples/feast/README.md b/examples/feast/README.md
index 06f88cb1..6dfe2064 100644
--- a/examples/feast/README.md
+++ b/examples/feast/README.md
@@ -1,15 +1,15 @@
-# Hamilton + Feast
+# Apache Hamilton + Feast
-In this example, we're going to show you how Hamilton can help you structure your Feast repository and bring tighter coupling between your feature transformations (code) and feature store (data).
+In this example, we're going to show you how Apache Hamilton can help you structure your Feast repository and bring tighter coupling between your feature transformations (code) and feature store (data).
- **Feast** is a feature store, which is an ML-specific stack component, that helps store and serve features (offline vs. online, batch vs. stream). It keeps a registry of features scattered across storage sources (database, data warehouse, streaming, etc.) and facilitates data retrieval and joining. Features need to be computed separately, typically in an SQL pipeline or a Python dataframe library and then be pushed to the Feast feature store([Feast FAQ](https://feast.dev/)).
-- **Hamilton** is a data transformation micro-framework. It helps one write Python code that is modular and reusable, and that expresses a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of execution. It was initially developed for large dataframes with hundreds of columns for machine learning while preserving strong lineage capabilities ([high-level comparison](https://hamilton.dagworks.io/en/latest/)).
+- **Apache Hamilton** is a data transformation micro-framework. It helps one write Python code that is modular and reusable, and that expresses a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of execution. It was initially developed for large dataframes with hundreds of columns for machine learning while preserving strong lineage capabilities ([high-level comparison](https://hamilton.dagworks.io/en/latest/)).
## File organization
-- `/default_feature_store` is the quickstart example you can generate by calling `feast init default`. It is presented here as a reference point to compare with Hamilton + Feast alternatives.
-- `/simple_feature_store` is a 1-to-1 reimplementation of `/default_feature_store`. You will notice that adding Hamilton helps make explicit the dependencies between Feast objects therefore increasing readability and maintainability.
-- `/integration_feature_store` extends the `/simple_feature_store` example by adding the feature transformation code using Hamilton and directly integrating with Feast.
-- `retrieval.ipynb` in `/integration_feature_store/feature_repo` shows how to retrieve features from Feast and highlights the benefits of end-to-end visibility with Hamilton.
+- `/default_feature_store` is the quickstart example you can generate by calling `feast init default`. It is presented here as a reference point to compare with Apache Hamilton + Feast alternatives.
+- `/simple_feature_store` is a 1-to-1 reimplementation of `/default_feature_store`. You will notice that adding Apache Hamilton helps make explicit the dependencies between Feast objects therefore increasing readability and maintainability.
+- `/integration_feature_store` extends the `/simple_feature_store` example by adding the feature transformation code using Apache Hamilton and directly integrating with Feast.
+- `retrieval.ipynb` in `/integration_feature_store/feature_repo` shows how to retrieve features from Feast and highlights the benefits of end-to-end visibility with Apache Hamilton.
## Learn more about Feast
- Hands-on workshop: https://github.com/feast-dev/feast-workshop
diff --git a/examples/feast/default_feature_store/README.md b/examples/feast/default_feature_store/README.md
index 4370b5c3..1544ff1c 100644
--- a/examples/feast/default_feature_store/README.md
+++ b/examples/feast/default_feature_store/README.md
@@ -1,5 +1,5 @@
# Context
-Anything in the directory `/default_feature_store`, including the rest of this `README` was generated with the CLI command `feast init default_feature_store`, which creates a Feast quickstart example. This example can provide useful information to new Feast users and serve as a comparison point to integrations with Hamilton. The directory `/simple_feature_store` is reimplements 1-to-1 the functionalities from this directory.
+Anything in the directory `/default_feature_store`, including the rest of this `README` was generated with the CLI command `feast init default_feature_store`, which creates a Feast quickstart example. This example can provide useful information to new Feast users and serve as a comparison point to integrations with Apache Hamilton. The directory `/simple_feature_store` is reimplements 1-to-1 the functionalities from this directory.
# Feast Quickstart
diff --git a/examples/feature_engineering/feature_engineering_multiple_contexts/README.md b/examples/feature_engineering/feature_engineering_multiple_contexts/README.md
index db41d67b..6e6f8c30 100644
--- a/examples/feature_engineering/feature_engineering_multiple_contexts/README.md
+++ b/examples/feature_engineering/feature_engineering_multiple_contexts/README.md
@@ -2,15 +2,15 @@
What is feature engineering? It's the process of transforming data for input to a "model".
-To make models better, it's common to perform and try a lot of "transforms". This is where Hamilton comes in.
-Hamilton allows you to:
+To make models better, it's common to perform and try a lot of "transforms". This is where Apache Hamilton comes in.
+Apache Hamilton allows you to:
* write different transformations in a straightforward and formulaic manner
* keep them managed and versioned with computational lineage (if using something like git)
* has a great testing and documentation story
which allows you to sanely iterate, maintain, and determine what works best for your modeling domain.
-In this series of examples, we'll skip talking about the benefits of Hamilton here, and instead focus on how to use it
+In this series of examples, we'll skip talking about the benefits of Apache Hamilton here, and instead focus on how to use it
for feature engineering. But first, some context on what challenges you're likely to face with feature engineering
in general.
@@ -62,16 +62,16 @@ With (1) and (2) in mind, you can see that there are a lot of different dimensio
feature engineering processes. They have to connect with each other, and be flexible enough to support your specific
deployment needs.
-# Using Hamilton for Feature Engineering for Batch/Offline
+# Using Apache Hamilton for Feature Engineering for Batch/Offline
If you fall into **only** needing to deploy features for batch jobs, then stop right there. You don't need these examples,
as they are focused on how to bridge the gap between "offline" and "online" feature engineering. You should instead
browse the other examples like `data_quality`.
-# Using Hamilton for Feature Engineering for Batch/Offline and Online/Streaming
+# Using Apache Hamilton for Feature Engineering for Batch/Offline and Online/Streaming
These example scenarios here are for the people who have to deal with both batch and online feature engineering.
We provide two examples for two common scenarios that occur if you have this need. Note, the example code in these
-scenarios tries to be illustrative about how to think and frame using Hamilton. It contains minimal features so as to
+scenarios tries to be illustrative about how to think and frame using Apache Hamilton. It contains minimal features so as to
not overwhelm you, and leaves out some implementation details that you would need to fill in for your specific use case,
e.g. like fitting a model using the features, or where to store aggregate feature values, etc.
@@ -81,7 +81,7 @@ setting, as well as an online setting (e.g. synchronous request via FastAPI). Wh
happens is that the code for features is not shared, and results in two implementations
that result in subtle bugs and hard to maintain code.
-With this example series, we show how you can use Hamilton to:
+With this example series, we show how you can use Apache Hamilton to:
1. write a feature once. (scenarios 1 and 2)
2. leverage that feature code anywhere that python runs. e.g. in batch and online. (scenarios 1 and 2)
@@ -90,8 +90,8 @@ you can inject those values into your feature computation needs. (scenario 2)
The task that we're modeling here isn't that important, but if you must know, we're trying to predict the number of
hours of absence that an employee will have given some information about them; this is based off the `data_quality`
-example, which is based off of the [Metaflow+Hamilton example](https://outerbounds.com/blog/developing-scalable-feature-engineering-dags/),
-where Hamilton was used for the feature engineering process -- in that example only offline feature engineering was modeled.
+example, which is based off of the [Metaflow+Apache Hamilton example](https://outerbounds.com/blog/developing-scalable-feature-engineering-dags/),
+where Apache Hamilton was used for the feature engineering process -- in that example only offline feature engineering was modeled.
Assumptions we're using:
1. You have a fixed set of features that you want to compute for a model that you have determined as being useful a priori.
@@ -116,15 +116,15 @@ at prediction time to get the right feature computation to happen.
In this scenario we assume we are not passed in data, but need to fetch it ourselves as part of the online API request.
We will pretend to hit a feature store, that will provide us with the required data to compute the features for
-input to the model. This example shows one way to modularize your Hamilton code so that you can swap out the "source"
+input to the model. This example shows one way to modularize your Apache Hamilton code so that you can swap out the "source"
of the data. To simplify the example, we assume that we can get all the input data we need from a feature store, rather
than it also coming in via the request. Note: if using a feature store, which is effectively a cache, you might not need
-Hamilton on the online side, if, and only if, you can get all the data you need from the feature store, without needing
+Apache Hamilton on the online side, if, and only if, you can get all the data you need from the feature store, without needing
to perform any computations. In this situation, you would push compute features to the feature store from your offline
ETL process that creates features.
A good exercise would be to make note of the differences with this scenario (2) and scenario (1) in how they structure
-the code with Hamilton.
+the code with Apache Hamilton.
# What's next?
Jump into each directory and read the README, it'll explain how the example is set up and how things should work.
@@ -132,7 +132,7 @@ Jump into each directory and read the README, it'll explain how the example is s
# What are extensions/uses not shown here but we know you can do them
Here are two ideas that come to mind:
-1. Streaming settings. Given the examples, it should be clear how to make it possbile to use Hamilton in a streaming setting.
-2. How to ask Hamilton what features are needed as input to know what to request from the feature store. With tags, and
-querying the DAG at the start of the app, you could dynamically ask Hamilton what's required and then only go to the
+1. Streaming settings. Given the examples, it should be clear how to make it possbile to use Apache Hamilton in a streaming setting.
+2. How to ask Apache Hamilton what features are needed as input to know what to request from the feature store. With tags, and
+querying the DAG at the start of the app, you could dynamically ask Apache Hamilton what's required and then only go to the
feature store for that data. If this type of example would be of interest, let us know.
diff --git a/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_1/README.md b/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_1/README.md
index aad79ec4..2cb0e3c2 100644
--- a/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_1/README.md
+++ b/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_1/README.md
@@ -20,7 +20,7 @@ into the feature computation process. E.g. If you're getting started, these coul
is loaded much like the model, or queried from a database, etc. Though you'll want
to ensure these values match whatever values the model was trained with. If you need help here, join our slack and we're happy to help you figure this out!
-Here's a mental image of how things work and how they relate to the files/modules & Hamilton:
+Here's a mental image of how things work and how they relate to the files/modules & Apache Hamilton:
@@ -38,7 +38,7 @@ Important not, there are two aggregations features defined: `age_mean` and `age_
`age` column. These make sense to compute in an offline setting as you have all the data, but in an online setting where
you'd be performing inference, that doesn't makse sense. So for the online case, these computations be "overridden" in
`fastapi_server.py` with the values that were computed in the offline setting that you have stored (as mentioned above
-and below it's up to you how to store them/sync them). The nice thing in Hamilton is that we can also "tag" these two
+and below it's up to you how to store them/sync them). The nice thing in Apache Hamilton is that we can also "tag" these two
feature transforms with information to indicate to someone reading the code, that they should be overriden in the
online feature computation context.
diff --git a/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_2/README.md b/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_2/README.md
index 63c3063c..ccc8d20a 100644
--- a/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_2/README.md
+++ b/examples/feature_engineering/feature_engineering_multiple_contexts/scenario_2/README.md
@@ -33,7 +33,7 @@ Important not, there are two aggregations features defined: `age_mean` and `age_
you'd be performing inference, that doesn't makse sense. So for the online case, in this example, they are "omitted" from
being nodes in the online context via `@config.when_not_in`. Instead, they are populated via the `online_loader.py` which
goes to the feature store for them. `fastapi_server.py` then is a little simpler than in Scenario (1), as it just
-delegates all work to Hamilton for getting features.
+delegates all work to Apache Hamilton for getting features.
## online_loader.py
Contains logic to load raw data in an online context from a feature store. It also contains functions to pull
diff --git a/examples/feature_engineering/write_once_run_everywhere_blog_post/README.md b/examples/feature_engineering/write_once_run_everywhere_blog_post/README.md
index 992a516b..a5a24678 100644
--- a/examples/feature_engineering/write_once_run_everywhere_blog_post/README.md
+++ b/examples/feature_engineering/write_once_run_everywhere_blog_post/README.md
@@ -75,7 +75,7 @@ we have exposed mock functions. To adapt this to your (real-world) use-case, you
4. See the commit history on this example, and see what was changed to accommodate the streaming context. You'll want to do something similar.
-# Pictures of the Hamilton DAG
+# Pictures of the Apache Hamilton DAG
## Batch
## Online
diff --git a/examples/hamilton-tutorials/mpg-translation/README.md b/examples/hamilton-tutorials/mpg-translation/README.md
index ca4e4e39..c12bad92 100644
--- a/examples/hamilton-tutorials/mpg-translation/README.md
+++ b/examples/hamilton-tutorials/mpg-translation/README.md
@@ -1,10 +1,10 @@
# Translation exercise
-This is a tutorial that one can use to help people grok Hamilton.
+This is a tutorial that one can use to help people grok Apache Hamilton.
-The task is to translate the procedural code into Hamilton. That is:
+The task is to translate the procedural code into Apache Hamilton. That is:
-1. Translate either mpg_script.py or ProceduralMPGNotebook.ipynb into a Hamilton.
+1. Translate either mpg_script.py or ProceduralMPGNotebook.ipynb into a Apache Hamilton.
2. There are two file stubs to fill in: `mpg_pipeline.py` or `MPGSimple.ipynb`.
If you get stuck or want to see the solution, you can look at `mpg_pipeline_target.py` or `MPGSimpleTarget.ipynb`.
@@ -17,7 +17,7 @@ To get set up:
1. Clone this repository.
2. Go into the `examples/hamilton-tutorials/mpg-translation` directory.
-2. Install Hamilton. `pip install - requirements.txt`
+2. Install Apache Hamilton. `pip install - requirements.txt`
3. Write code.
4. Excute via notebook or command line.
diff --git a/examples/hamilton_ui/README.md b/examples/hamilton_ui/README.md
index d834c7a6..6fe4a374 100644
--- a/examples/hamilton_ui/README.md
+++ b/examples/hamilton_ui/README.md
@@ -1,6 +1,6 @@
-# Hamilton UI - Machine learning example
+# Apache Hamilton UI - Machine learning example
-Learn how to use the `HamiltonTracker` and the Hamilton UI to track a simple machine learning pipeline.
+Learn how to use the `HamiltonTracker` and the Apache Hamilton UI to track a simple machine learning pipeline.
It also illustrates the following notions:
@@ -10,9 +10,9 @@ It also illustrates the following notions:
## Getting started
-### Install the Hamilton UI
+### Install the Apache Hamilton UI
-First, you need to have the Hamilton UI running. You can either `pip install` the Hamilton UI (recommended) or run it as a Docker container.
+First, you need to have the Apache Hamilton UI running. You can either `pip install` the Apache Hamilton UI (recommended) or run it as a Docker container.
#### Local Install
Install the Python dependencies:
@@ -20,7 +20,7 @@ Install the Python dependencies:
```bash
pip install "sf-hamilton[ui,sdk]"
```
-then launch the Hamilton UI server:
+then launch the Apache Hamilton UI server:
```bash
hamilton ui
# python -m hamilton.cli.__main__ ui # on windows
@@ -40,7 +40,7 @@ See [this video](https://youtu.be/DPfxlTwaNsM) for a walkthrough.
### Execute and track the pipeline
-Now that you have the Hamilton UI running, open another terminal tab to:
+Now that you have the Apache Hamilton UI running, open another terminal tab to:
1. Ensure you have the right python dependencies installed.
```bash
@@ -48,7 +48,7 @@ cd hamilton/examples/hamilton_ui
pip install -r requirements.txt
```
-2. Run the `run.py` script. Providing the username and project ID to be able to log to the Hamilton UI.
+2. Run the `run.py` script. Providing the username and project ID to be able to log to the Apache Hamilton UI.
```bash
python run.py --username <username> --project_id <project_id>
```
@@ -57,12 +57,12 @@ Once you've run that, run this:
python run.py --username <username> --project_id <project_id> --load-from-parquet
```
-3. Explore results in the Hamilton UI. Find your project under http://localhost:8242/dashboard/projects.
+3. Explore results in the Apache Hamilton UI. Find your project under http://localhost:8242/dashboard/projects.
## Things to try:
-1. Place an error in the code and see how it shows up in the Hamilton UI. e.g. `raise ValueError("I'm an error")`.
+1. Place an error in the code and see how it shows up in the Apache Hamilton UI. e.g. `raise ValueError("I'm an error")`.
2. In `models.py` change `"data_set": source("data_set_v1"),` to `"data_set": source("data_set_v2"),`, along with
-what is requested in `run.py` (i.e. change/add saving `data_set_v2`) and see how the lineage changes in the Hamilton UI.
+what is requested in `run.py` (i.e. change/add saving `data_set_v2`) and see how the lineage changes in the Apache Hamilton UI.
3. Add a new feature and propagate it through the pipeline. E.g. add a new feature to `features.py` and then to a dataset.
-Execute it and then compare the data observed in the Hamilton UI against a prior run.
+Execute it and then compare the data observed in the Apache Hamilton UI against a prior run.
diff --git a/examples/hello_world/README.md b/examples/hello_world/README.md
index 940b37ed..d2fcf27c 100644
--- a/examples/hello_world/README.md
+++ b/examples/hello_world/README.md
@@ -1,4 +1,4 @@
-# Classic Hamilton Hello World
+# Classic Apache Hamilton Hello World
In this example we show you how to create a simple hello world dataflow that
creates a dataframe as a result. It performs a series of transforms on the
@@ -8,8 +8,8 @@ File organization:
* `my_functions.py` houses the logic that we want to compute. Note how the functions are named, and what input
parameters they require. That is how we create a DAG modeling the dataflow we want to happen.
-* `my_script.py` houses how to get Hamilton to create the DAG and exercise it with some inputs.
-* `my_notebook.ipynb` houses how one might iterate in a notebook environment and provide a way to inline define Hamilton
+* `my_script.py` houses how to get Apache Hamilton to create the DAG and exercise it with some inputs.
+* `my_notebook.ipynb` houses how one might iterate in a notebook environment and provide a way to inline define Apache Hamilton
functions and add them to the DAG constructed. To be clear, it is not used by `my_script.py`, but showing an alternate path
to running/developing things.
diff --git a/examples/ibis/feature_engineering/README.md b/examples/ibis/feature_engineering/README.md
index 75aeed32..0e5eef37 100644
--- a/examples/ibis/feature_engineering/README.md
+++ b/examples/ibis/feature_engineering/README.md
@@ -1,8 +1,8 @@
-# Ibis + Hamilton
+# Ibis + Apache Hamilton
-[Ibis](https://ibis-project.org/) is a portable dataframe library to write procedural data transformations in Python and be able to execute them directly on various SQL backends (DuckDB, Snowflake, Postgres, Flink, see [full list](https://ibis-project.org/support_matrix)). Hamilton provides a declarative way to define testable, modular, self-documenting dataflows, that encode lineage and metadata.
+[Ibis](https://ibis-project.org/) is a portable dataframe library to write procedural data transformations in Python and be able to execute them directly on various SQL backends (DuckDB, Snowflake, Postgres, Flink, see [full list](https://ibis-project.org/support_matrix)). Apache Hamilton provides a declarative way to define testable, modular, self-documenting dataflows, that encode lineage and metadata.
-In this example, we'll show how to get started with creating feature transformations and training a machine learning model. You'll learn about the basics of Ibis and IbisML and how they integrate with Hamilton.
+In this example, we'll show how to get started with creating feature transformations and training a machine learning model. You'll learn about the basics of Ibis and IbisML and how they integrate with Apache Hamilton.
@@ -21,7 +21,7 @@ Follow these steps to get the example working:
pip install -r requirements.txt
```
-3. execute the Hamilton feature engineering dataflow at the table or column level
+3. execute the Apache Hamilton feature engineering dataflow at the table or column level
```script
python run.py --level [table, column]
@@ -29,10 +29,10 @@ Follow these steps to get the example working:
# Files
- `table_dataflow.py` and `column_dataflow.py` include the same Ibis feature engineering dataflow, but with different level of granularity
-- `tables.png` and `columns.png` were generated by Hamilton directly from the code.
+- `tables.png` and `columns.png` were generated by Apache Hamilton directly from the code.
- `ibis_feature_set.png` was generated by Ibis. It describes the atomic data transformations executed by the expression.
# Resources
-- [Learn more about Hamilton + Ibis](https://hamilton.dagworks.io/en/latest/integrations/ibis/)
+- [Learn more about Apache Hamilton + Ibis](https://hamilton.dagworks.io/en/latest/integrations/ibis/)
- [Ibis documentation](https://ibis-project.org/)
diff --git a/examples/ibis/jaffle_shop/README.md b/examples/ibis/jaffle_shop/README.md
index 2bd92f10..dcb59379 100644
--- a/examples/ibis/jaffle_shop/README.md
+++ b/examples/ibis/jaffle_shop/README.md
@@ -2,17 +2,17 @@
This repository is a reimplementation of the canonical [dbt example jaffle_shop](https://github.com/dbt-labs/jaffle_shop). It illustrates data transformations for an ecommerce store.
-Data transformations are implemented using the Python library [Ibis](https://ibis-project.org/) which allows to define SQL operations that works across backends. By default, it uses [duckdb](https://duckdb.org/) for local development. Hamilton + Ibis provides a Python-centric alternative to dbt ([Learn more](https://hamilton.dagworks.io/en/latest/integrations/ibis/)).
+Data transformations are implemented using the Python library [Ibis](https://ibis-project.org/) which allows to define SQL operations that works across backends. By default, it uses [duckdb](https://duckdb.org/) for local development. Apache Hamilton + Ibis provides a Python-centric alternative to dbt ([Learn more](https://hamilton.dagworks.io/en/latest/integrations/ibis/)).
## Content
-The content and structure aims to match the [original dbt `jaffle_shop`](https://github.com/dbt-labs/jaffle_shop/tree/main) example. On the other hand, Hamilton is just a Python library and is flexible regarding project structure.
+The content and structure aims to match the [original dbt `jaffle_shop`](https://github.com/dbt-labs/jaffle_shop/tree/main) example. On the other hand, Apache Hamilton is just a Python library and is flexible regarding project structure.
- `data/`: samples data; equivalent to `seeds/` in the dbt repo.
- `dataflows/staging` load raw data and rename columns to avoid naming conflicts; equivalent to `models/staging/` in the dbt repo.
- `dataflows/customer_flow.py` and `dataflows/order_flow.py` define data transformations; equivalent to `models/customers.sql` and `models/orders.sql` in the dbt repo.
- `run.py` specify where to load data from and how to execute dataflows.
- Generally, you'll notice Hamilton aims to reduce the sprawl of configurations (`.yaml`) and documentation (`.md`). Instead, it uses docstrings, type hints, or Python object to couple them to with your code (`.py`).
+ Generally, you'll notice Apache Hamilton aims to reduce the sprawl of configurations (`.yaml`) and documentation (`.md`). Instead, it uses docstrings, type hints, or Python object to couple them to with your code (`.py`).
## Set up
1. create and activate virtual environment
@@ -26,7 +26,7 @@ The content and structure aims to match the [original dbt `jaffle_shop`](https:/
pip install -r requirements.txt
```
-3. execute the Hamilton dataflow
+3. execute the Apache Hamilton dataflow
```script
python run.py
diff --git a/examples/ibisml/README.md b/examples/ibisml/README.md
index d6279e8e..2af3a2f6 100644
--- a/examples/ibisml/README.md
+++ b/examples/ibisml/README.md
@@ -1 +1 @@
-# IbisML + Hamilton [WIP]
+# IbisML + Apache Hamilton [WIP]
diff --git a/examples/jupyter_notebook_magic/README.md b/examples/jupyter_notebook_magic/README.md
index 0ec2d74f..e6d866ac 100644
--- a/examples/jupyter_notebook_magic/README.md
+++ b/examples/jupyter_notebook_magic/README.md
@@ -1,8 +1,8 @@
-# Hamilton notebook extension
+# Apache Hamilton notebook extension
-One of the best part about notebooks is the ability to execute and immediately inspect results. They provide a "read-eval-print" loop (REPL) coding experience. However, the way Hamilton separates dataflow definition (functions in a module) from execution (building and executing a driver) creates an extra step that can slowdown this loop.
+One of the best part about notebooks is the ability to execute and immediately inspect results. They provide a "read-eval-print" loop (REPL) coding experience. However, the way Apache Hamilton separates dataflow definition (functions in a module) from execution (building and executing a driver) creates an extra step that can slowdown this loop.
-We built the Hamilton notebook extension to tighten that loop and even give a better experience than the core notebook experience!
+We built the Apache Hamilton notebook extension to tighten that loop and even give a better experience than the core notebook experience!
For a [video overview click here](https://www.youtube.com/watch?v=Z3ZT2ur2jg8&t=288s).
diff --git a/examples/kedro/README.md b/examples/kedro/README.md
index 4ca3eeba..6d00a0b0 100644
--- a/examples/kedro/README.md
+++ b/examples/kedro/README.md
@@ -1,12 +1,12 @@
-# Kedro & Hamilton
+# Kedro & Apache Hamilton
-This repository compares how to build dataflows with Kedro and Hamilton.
+This repository compares how to build dataflows with Kedro and Apache Hamilton.
-> see the [side-by-side comparison](https://hamilton.dagworks.io/en/latest/code-comparisons/kedro/) in the Hamilton documentation
+> see the [side-by-side comparison](https://hamilton.dagworks.io/en/latest/code-comparisons/kedro/) in the Apache Hamilton documentation
## Content
- `kedro-code/` includes code from the [Kedro Spaceflight tutorial](https://docs.kedro.org/en/stable/tutorial/tutorial_template.html).
-- `hamilton-code/` is a refactor of `kedro-code/` using the Hamilton library.
-- `kedro-plugin/` showcases Hamilton plugins to integrate with the Kedro framework.
+- `hamilton-code/` is a refactor of `kedro-code/` using the Apache Hamilton library.
+- `kedro-plugin/` showcases Apache Hamilton plugins to integrate with the Kedro framework.
-Each directory contains a `README` with instructions on how to run the code. We suggest going through the Kedro code first, and then read the Hamilton refactor.
+Each directory contains a `README` with instructions on how to run the code. We suggest going through the Kedro code first, and then read the Apache Hamilton refactor.
diff --git a/examples/kedro/hamilton-code/README.md b/examples/kedro/hamilton-code/README.md
index e1702291..235397d7 100644
--- a/examples/kedro/hamilton-code/README.md
+++ b/examples/kedro/hamilton-code/README.md
@@ -1,13 +1,13 @@
-# Hamilton code
+# Apache Hamilton code
-> Hamilton dataflow
+> Apache Hamilton dataflow
## File structure
-The Hamilton refactor is composed of a few files:
-- `data_processing.py` and `data_science.py` contains regular Python functions to define the Hamilton dataflow. This is equivalent to Kedro's `pipeline.py` **and** `nodes.py` files.
+The Apache Hamilton refactor is composed of a few files:
+- `data_processing.py` and `data_science.py` contains regular Python functions to define the Apache Hamilton dataflow. This is equivalent to Kedro's `pipeline.py` **and** `nodes.py` files.
- `run.py` contains the "driver code" to load and execute the dataflow. There's no direct equivalent in the Kedro tutorial since it prefers using the CLI for execution.
-- `noteboks/interactive.ipynb` contains the "driver code", similar to `run.py`, but uses [Hamilton Jupyter Magics](https://hamilton.dagworks.io/en/latest/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic) to define the dataflow interactily in a notebook.
+- `noteboks/interactive.ipynb` contains the "driver code", similar to `run.py`, but uses [Apache Hamilton Jupyter Magics](https://hamilton.dagworks.io/en/latest/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic) to define the dataflow interactily in a notebook.
- `tests/test_dataflow.py` includes tests equivalent to `tests/pipelines/data_science/test_pipeline.py` in the Kedro code.
## Instructions
@@ -16,7 +16,7 @@ The Hamilton refactor is composed of a few files:
python -m venv venv && . venv/bin/active
```
-2. Install requirements for the Hamilton code
+2. Install requirements for the Apache Hamilton code
```console
pip install -r requirements.txt
```
@@ -37,6 +37,6 @@ The Hamilton refactor is composed of a few files:
```
## Going further
-- Learn the basics of Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
+- Learn the basics of Apache Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
- Visit [tryhamilton.dev](tryhamilton.dev) for an interactive tutorial in your browser
- Visit the [DAGWorks blog](https://blog.dagworks.io/) for more detailed guides
diff --git a/examples/kedro/kedro-plugin/README.md b/examples/kedro/kedro-plugin/README.md
index 8d43cd3c..d6aa4b65 100644
--- a/examples/kedro/kedro-plugin/README.md
+++ b/examples/kedro/kedro-plugin/README.md
@@ -1,4 +1,4 @@
# Kedro plugin
## Content
-- `kedro_to_hamilton.ipynb` contains a tutorial on how to execute your Kedro `Pipeline` using Hamilton, track your execution in the Hamilton UI, and use the Kedro materializers to load & save data.
+- `kedro_to_hamilton.ipynb` contains a tutorial on how to execute your Kedro `Pipeline` using Apache Hamilton, track your execution in the Apache Hamilton UI, and use the Kedro materializers to load & save data.
diff --git a/examples/lineage/README.md b/examples/lineage/README.md
index 2be04da7..d96541bd 100644
--- a/examples/lineage/README.md
+++ b/examples/lineage/README.md
@@ -1,16 +1,16 @@
-# Lineage + Hamilton
+# Lineage + Apache Hamilton
-<div style="text-align: center;">Example lineage graph generated by Hamilton when you write Hamilton code.</div>
+<div style="text-align: center;">Example lineage graph generated by Apache Hamilton when you write Apache Hamilton code.</div>
-Here we showcase Hamilton's lineage abilities. We will use the Titanic
+Here we showcase Apache Hamilton's lineage abilities. We will use the Titanic
data set to model a hypothetical company set up where there is data engineering and data science team collaborating together.
If you want to see code and what it does we invite you to jump straight into browsing the [lineage_snippets](https://github.com/apache/hamilton/blob/main/examples/lineage/lineage_snippets.ipynb) notebook.
For those coming from the lineage blog post, you can find the code shown in [lineage_script.py](https://github.com/apache/hamilton/blob/main/examples/lineage/lineage_script.py).
For those who want to continue, let's first talk about some common problems people encounter, then more formally frame what we mean by
-lineage, and then explain how Hamilton's lineage capabilities help you solve common problems encountered with Data and Machine Learning.
+lineage, and then explain how Apache Hamilton's lineage capabilities help you solve common problems encountered with Data and Machine Learning.
Note: a quick word on terminology, we use `function` and `node` interchangeably because you write functions, that get turned
into nodes that are modeled in a DAG. So if you see `function` or `node` they mean the same thing effectively.
@@ -52,16 +52,16 @@ data transformations and model training processes.
- Collaboration: Lineage allows different stakeholders to understand the data's history, facilitating collaboration
between teams working on different parts of a project.
-## Hamilton's Lineage Capabilities
-*Good news*: Hamilton provides a lot of the functionality needed for storing lineage and asking questions of it. Here
-we'll walk through a few features of Hamilton that will help answer and empower teams targeting the four points above:
+## Apache Hamilton's Lineage Capabilities
+*Good news*: Apache Hamilton provides a lot of the functionality needed for storing lineage and asking questions of it. Here
+we'll walk through a few features of Apache Hamilton that will help answer and empower teams targeting the four points above:
- reproducibility
- auditing and compliance
- troubleshooting and debugging
- collaboration
### Lineage as Code
-To start, Hamilton by design, encodes lineage information as code. This means, as you write each Hamilton function,
+To start, Apache Hamilton by design, encodes lineage information as code. This means, as you write each Apache Hamilton function,
you are encoding lineage information required to compute it, i.e. by specifying a dataflow you have in effect,
specified lineage! This means, as you write your code and commit to, for example a version control system,
you have a record of how computation should happen. A huge benefit of this, is when the code changes, so does this
@@ -70,36 +70,36 @@ information -- all without you having to manage a separate system!
TL;DR:
> No need for a separate system to store lineage information, it's already in your code!
-**The one thing you need to manage** is that Hamilton does not store information on artifacts it produces. If you're
-producing a dataframe, a model, or some other object that your Hamilton code computes, you need to store that. The good
+**The one thing you need to manage** is that Apache Hamilton does not store information on artifacts it produces. If you're
+producing a dataframe, a model, or some other object that your Apache Hamilton code computes, you need to store that. The good
news is, you're likely already doing this! **But**, you're probably not storing the lineage information that produced
-that artifact, and that's where Hamilton comes in.
+that artifact, and that's where Apache Hamilton comes in.
-For each artifact you produce, you just need to associate the Hamilton DAG that produced it. This is as simple as
-storing the git SHA of the code and snapshot of configuration that created your Hamilton DAG, so you can retrieve the
+For each artifact you produce, you just need to associate the Apache Hamilton DAG that produced it. This is as simple as
+storing the git SHA of the code and snapshot of configuration that created your Apache Hamilton DAG, so you can retrieve the
code and ask questions of it. Adding this extra information is easy since most destinations to store artifacts allow for
extra metadata to be stored, e.g. from MLFlow for models, Snowflake as table metadata for tables/dataframes, to flat
files on S3.
-Let's explain how using Hamilton helps get at the four points above.
+Let's explain how using Apache Hamilton helps get at the four points above.
### Reproducibility
-By writing Hamilton code and connecting it with a version control system (e.g. git) you have by definition written
-code that can reproduce results. This is because Hamilton DAGs are deterministic. The version control system is a
+By writing Apache Hamilton code and connecting it with a version control system (e.g. git) you have by definition written
+code that can reproduce results. This is because Apache Hamilton DAGs are deterministic. The version control system is a
straightforward way to store evolutions of your code and configuration, and therefore your DAGs.
By versioning code, you are therefore versioning lineage information. This means you can go back in time and ask
questions about the past. For example, you can ask what the lineage information was for a model that was trained at
a specific point in time. This is as simple as checking out the git SHA of the code that produced the model, and
-asking Hamilton to visualize (e.g. see [`visualize_execution()`](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.visualize_execution)), the DAG and ask questions of it.
+asking Apache Hamilton to visualize (e.g. see [`visualize_execution()`](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.visualize_execution)), the DAG and ask questions of it.
### Auditing and Compliance
The [`@tag`](https://hamilton.dagworks.io/en/latest/reference/decorators/tag) and (`@tag_outputs`) feature allows
-you to annotate your functions with metadata. No extra YAML file to manage, just directly together with your Hamilton code.
-This means you can tag functions with information such as "PII", "GDPR", "HIPAA", etc. and then ask Hamilton to return
+you to annotate your functions with metadata. No extra YAML file to manage, just directly together with your Apache Hamilton code.
+This means you can tag functions with information such as "PII", "GDPR", "HIPAA", etc. and then ask Apache Hamilton to return
nodes with certain tags, e.g. get me all my "sources", or "what is PII, and what consumes it?", etc.
-The [Hamilton Driver](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver)
+The [Apache Hamilton Driver](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver)
has a lot of functions that allow you to ask questions of your DAGs to make (1) easier. The driver code can be run in
a CI/CD system, or as part of a reporting/auditing pipeline. For example, you can ask:
@@ -113,9 +113,9 @@ With these three functions you can the find functions with specific tags and the
The good news is that what is great for reproducibility, auditing and reproducibility, is also great for
troubleshooting and debugging.
-Debugging is methodical and procedural with Hamilton. The way functions are written
+Debugging is methodical and procedural with Apache Hamilton. The way functions are written
and executed mean that one can easily walk through just the part of the DAG of interest to debug an issue.
-To help with this, Hamilton has various methods to visualize lineage so you can more easily see what you're walking through connects to:
+To help with this, Apache Hamilton has various methods to visualize lineage so you can more easily see what you're walking through connects to:
* [`display_all_functions()`](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.display_all_functions)
* [`display_downstream_of()`](https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.display_downstream_of)
@@ -125,26 +125,26 @@ To help with this, Hamilton has various methods to visualize lineage so you can
### Collaboration
When any organization scales, or has personnel changes, it's important to have a system that helps people get up to
-in a self-service manner. Hamilton's lineage as code approach means that new team members can easily get up to speed
-because functions are written in a standard way, and the lineage information is encoded in the code itself. The Hamilton
+in a self-service manner. Apache Hamilton's lineage as code approach means that new team members can easily get up to speed
+because functions are written in a standard way, and the lineage information is encoded in the code itself. The Apache Hamilton
Driver code enables one to ask questions of the DAGs, and therefore the code, to get up to speed quickly.
-## Recipe for using Hamilton's Lineage Capabilities
+## Recipe for using Apache Hamilton's Lineage Capabilities
-At a high level, the recipe for utilizing Hamilton's lineage capabilities is as follows:
+At a high level, the recipe for utilizing Apache Hamilton's lineage capabilities is as follows:
-1. Write Hamilton code.
+1. Write Apache Hamilton code.
2. Use [`@tag`](https://hamilton.dagworks.io/en/latest/reference/decorators/tag) and
[`@tag_outputs`](https://hamilton.dagworks.io/en/latest/reference/decorators/tag) to
annotate functions.
-3. Instantiate a Hamilton Driver, it'll then have a representation of how data and compute flow as defined by your
-Hamilton code. The Driver object can then emit/provide information on lineage!
-4. If you store Hamilton code with your version control system, you can then go back in time to understand how lineage
+3. Instantiate a Apache Hamilton Driver, it'll then have a representation of how data and compute flow as defined by your
+Apache Hamilton code. The Driver object can then emit/provide information on lineage!
+4. If you store Apache Hamilton code with your version control system, you can then go back in time to understand how lineage
changes over time, since it's encoded in code!
In code this should look something like the following:
-(1) and (2) write Hamilton code and annotate with `@tag` (and/or `@tag_outputs`).
+(1) and (2) write Apache Hamilton code and annotate with `@tag` (and/or `@tag_outputs`).
```python
@tag(owner="data-science", importance="production", artifact="model")
def fit_random_forest(
@@ -155,7 +155,7 @@ def fit_random_forest(
"""Returns a fit RF model."""
# ... contents of function not important ... code skipped for brevity
```
-(3) Instantiate a Hamilton Driver and ask questions of it.
+(3) Instantiate a Apache Hamilton Driver and ask questions of it.
```python
from hamilton import base
from hamilton import driver
@@ -199,7 +199,7 @@ To see more code, we invite you to:
1. Browse the modules to see what the functions are and what they're annotated with.
2. Browse either the [lineage_snippets](https://github.com/apache/hamilton/blob/main/examples/lineage/lineage_snippets.ipynb) notebook
or the [lineage_script](https://github.com/apache/hamilton/blob/main/examples/lineage/lineage_script.py)
-to see how to use the Hamilton Driver to ask questions of your DAGs.
+to see how to use the Apache Hamilton Driver to ask questions of your DAGs.
3. We invite you to then go back in time, by checking out this repository and checking out an older commit and
re-running the script or notebook and seeing how things change. The command to "go back in time" would be:
```bash
diff --git a/examples/materialization/README.md b/examples/materialization/README.md
index 3a6c73e8..6e411bf5 100644
--- a/examples/materialization/README.md
+++ b/examples/materialization/README.md
@@ -1,6 +1,6 @@
# Materialization
-Hamilton's driver allows for ad-hoc materialization. This enables you to take a DAG you already have,
+Apache Hamilton's driver allows for ad-hoc materialization. This enables you to take a DAG you already have,
and save your data to a set of custom locations/url.
Note that these materializers are _isomorphic_ in nature to the
diff --git a/examples/materialization/datasaver_dataloader_example/README.md b/examples/materialization/datasaver_dataloader_example/README.md
index 4a521fc1..1d28c95b 100644
--- a/examples/materialization/datasaver_dataloader_example/README.md
+++ b/examples/materialization/datasaver_dataloader_example/README.md
@@ -5,7 +5,7 @@ This example shows you can use the
annotated functions that save and load data so that they
can also return metadata.
-This metadata is then exposed in the Hamilton UI / inspectable
+This metadata is then exposed in the Apache Hamilton UI / inspectable
by other adapters.
To run this example, you can use the following command:
@@ -14,8 +14,8 @@ To run this example, you can use the following command:
```bash
pip install -r requirements.txt
```
-2. Have the Hamilton UI set up. See [UI docs](https://hamilton.dagworks.io/en/latest/hamilton-ui/) for instructions.
-3. You have modified the run.py / notebook to log to the right Hamilton UI project.
+2. Have the Apache Hamilton UI set up. See [UI docs](https://hamilton.dagworks.io/en/latest/hamilton-ui/) for instructions.
+3. You have modified the run.py / notebook to log to the right Apache Hamilton UI project.
4. Run the example:
```bash
diff --git a/examples/mlflow/README.md b/examples/mlflow/README.md
index 4891302b..93a8b277 100644
--- a/examples/mlflow/README.md
+++ b/examples/mlflow/README.md
@@ -1,14 +1,14 @@
-# MLFLow plugin for Hamilton
+# MLFLow plugin for Apache Hamilton
[MLFlow](https://mlflow.org/) is an open-source Python framework for experiment tracking. It allows data science teams to store results, artifacts (machine learning models, figures, tables), and metadata in a principled way when executing data pipelines.
-The MLFlow plugin for Hamilton includes two sets of features:
+The MLFlow plugin for Apache Hamilton includes two sets of features:
- Save and load machine learning models with the `MLFlowModelSaver` and `MLFlowModelLoader` materializers
- Automatically track data pipeline results in MLFlow with the `MLFlowTracker`.
-This pairs nicely with the `HamiltonTracker` and the [Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/) which gives you a way to explore your pipeline code, attributes of the artifacts produced, and execution observability.
+This pairs nicely with the `HamiltonTracker` and the [Apache Hamilton UI](https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/) which gives you a way to explore your pipeline code, attributes of the artifacts produced, and execution observability.
-We're working on better linking Hamilton "projects" with MLFlow "experiments" and runs from both projects.
+We're working on better linking Apache Hamilton "projects" with MLFlow "experiments" and runs from both projects.
## Instructions
1. Create a virtual environment and activate it
@@ -16,7 +16,7 @@ We're working on better linking Hamilton "projects" with MLFlow "experiments" an
python -m venv venv && . venv/bin/active
```
-2. Install requirements for the Hamilton code
+2. Install requirements for the Apache Hamilton code
```console
pip install -r requirements.txt
```
@@ -30,6 +30,6 @@ We're working on better linking Hamilton "projects" with MLFlow "experiments" an
```
## Going further
-- Learn the basics of Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
+- Learn the basics of Apache Hamilton via the `Concepts/` [documentation section](https://hamilton.dagworks.io/en/latest/concepts/node/)
- Visit [tryhamilton.dev](tryhamilton.dev) for an interactive tutorial in your browser
- Visit the [DAGWorks blog](https://blog.dagworks.io/) for more detailed guides
diff --git a/examples/model_examples/README.md b/examples/model_examples/README.md
index 10db9509..3d1e5fa4 100644
--- a/examples/model_examples/README.md
+++ b/examples/model_examples/README.md
@@ -1,6 +1,6 @@
-# Creating model training/predict workflows with Hamilton
+# Creating model training/predict workflows with Apache Hamilton
-Hamilton is a general purpose dataflow framework. So you can
+Apache Hamilton is a general purpose dataflow framework. So you can
model pretty much any workflow you'd like to compute.
Here we'll house dataflows that show how to build ML models.
diff --git a/examples/model_examples/modular_example/README.md b/examples/model_examples/modular_example/README.md
index ef48d4de..c0076c04 100644
--- a/examples/model_examples/modular_example/README.md
+++ b/examples/model_examples/modular_example/README.md
@@ -1,7 +1,7 @@
# Modular pipeline example
In this example we show how you can compose a pipeline from multiple modules.
-This is a common pattern in Hamilton, where you can define a module that encapsulates
+This is a common pattern in Apache Hamilton, where you can define a module that encapsulates
a set of "assets" and then use that module in a parameterized manner.
The use case here is that:
diff --git a/examples/model_examples/scikit-learn/README.md b/examples/model_examples/scikit-learn/README.md
index a877a18d..21e42532 100644
--- a/examples/model_examples/scikit-learn/README.md
+++ b/examples/model_examples/scikit-learn/README.md
@@ -1,7 +1,7 @@
-# Using Hamilton for ML dataflows
+# Using Apache Hamilton for ML dataflows
Here we have a simple example showing how you can
-write a ML training and evaluation workflow with Hamilton.
+write a ML training and evaluation workflow with Apache Hamilton.
`pip install scikit-learn` or `pip install -r requirements.txt` to get the right python dependencies
to run this example.
diff --git a/examples/model_examples/time-series/README.md b/examples/model_examples/time-series/README.md
index e7de11d3..3a91ed20 100644
--- a/examples/model_examples/time-series/README.md
+++ b/examples/model_examples/time-series/README.md
@@ -1,7 +1,7 @@
# Kaggle M5 Forecasting
Source - https://www.kaggle.com/c/m5-forecasting-accuracy.
-This is an example of how one might use Hamilton using the M5 Forecasting Kaggle
+This is an example of how one might use Apache Hamilton using the M5 Forecasting Kaggle
challenge as an example. The code here is based on Ratan Rohith's notebook
that can be [found here](https://www.kaggle.com/code/ratan123/m5-forecasting-lightgbm-with-timeseries-splits).
The notebook is a beginner level notebook that does some feature engineering, and uses LightGBM to fit
@@ -18,7 +18,7 @@ you will need to log in to Kaggle to download the data.
You can even run this example in Google Colab:
[
-](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/model_examples/time-series/Hamilton-TimeSeriesmodel.ipynb)
+](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/model_examples/time-series/Apache Hamilton-TimeSeriesmodel.ipynb)
diff --git a/examples/module_overrides/README.md b/examples/module_overrides/README.md
index cff0203b..932ca07e 100644
--- a/examples/module_overrides/README.md
+++ b/examples/module_overrides/README.md
@@ -32,7 +32,7 @@ def weighted_average(data:pd.Series, weight:int)->pd.Series:
```
-In this case Hamilton will raise an error since we cannot have two same named functions in the DAG.
+In this case Apache Hamilton will raise an error since we cannot have two same named functions in the DAG.
We have a handy flag for you to allow the later imported module to overwrite the previous same-named functions:
@@ -44,4 +44,4 @@ dr = (
)
```
-which will tell Hamilton to use ``module_B.weighted_average()`` for the node and ignore the same-named function from ``module_A``.
+which will tell Apache Hamilton to use ``module_B.weighted_average()`` for the node and ignore the same-named function from ``module_A``.
diff --git a/examples/numpy/air-quality-analysis/README.md b/examples/numpy/air-quality-analysis/README.md
index c0088d1f..4ac67336 100644
--- a/examples/numpy/air-quality-analysis/README.md
+++ b/examples/numpy/air-quality-analysis/README.md
@@ -3,9 +3,9 @@
This is taken from the numpy tutorial https://github.com/numpy/numpy-tutorials/blob/main/content/tutorial-air-quality-analysis.md.
# analysis_flow.py
-Is where the analysis steps are defined as Hamilton functions.
+Is where the analysis steps are defined as Apache Hamilton functions.
-Versus doing this analysis in a notebook, the strength of Hamilton here is in
+Versus doing this analysis in a notebook, the strength of Apache Hamilton here is in
forcing concise definitions and language around steps in the analysis -- and
then magically the analysis is pretty reusable / very easy to augment. E.g. add some
@config.when or split things into python modules to be swapped out, to extend the
@@ -27,5 +27,5 @@ You can even run this example in Google Colab:
# Caveat
-The code found here was copied and pasted, and then tweaked to run with Hamilton. If something from the modeling
+The code found here was copied and pasted, and then tweaked to run with Apache Hamilton. If something from the modeling
perspective isn't clear, please read https://github.com/numpy/numpy-tutorials/blob/main/content/tutorial-air-quality-analysis.md
diff --git a/examples/openlineage/README.md b/examples/openlineage/README.md
index a57dbaef..0adaa0f8 100644
--- a/examples/openlineage/README.md
+++ b/examples/openlineage/README.md
@@ -4,10 +4,10 @@ This is an example of how to use the OpenLineage adapter that can be used to sen
## Motivation
OpenLineage is an open standard for data lineage.
-With Hamilton you can read and write data, and with OpenLineage you can track the lineage of that data.
+With Apache Hamilton you can read and write data, and with OpenLineage you can track the lineage of that data.
## Steps
-1. Build your project with Hamilton.
+1. Build your project with Apache Hamilton.
2. Use one of the [materialization approaches](https://hamilton.dagworks.io/en/latest/concepts/materialization/) to surface metadata about what is loaded and saved.
3. Use the OpenLineage adapter to send metadata to an OpenLineage server.
diff --git a/examples/opentelemetry/README.md b/examples/opentelemetry/README.md
index 26fb763c..e98aad78 100644
--- a/examples/opentelemetry/README.md
+++ b/examples/opentelemetry/README.md
@@ -1,6 +1,6 @@
-# Hamilton + OpenTelemetry
+# Apache Hamilton + OpenTelemetry
You can use the `hamilton.plugins.OpenTelemetryTracer()` to send telemetry
-about your Hamilton dataflow.
+about your Apache Hamilton dataflow.
See [the example with LLMs and OpenLLMetry](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/observability/openllmetry) for more information.
diff --git a/examples/pandas/split-apply-combine/README.md b/examples/pandas/split-apply-combine/README.md
index 216b78ee..5ecca18c 100644
--- a/examples/pandas/split-apply-combine/README.md
+++ b/examples/pandas/split-apply-combine/README.md
@@ -2,7 +2,7 @@
This example demonstrates how to perform
a [split-apply-combine](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html) transformation using
-Hamilton.
+Apache Hamilton.
Many data analysis or processing involve one or more of the following steps:
diff --git a/examples/parallelism/README.md b/examples/parallelism/README.md
index cca2377a..95ca338f 100644
--- a/examples/parallelism/README.md
+++ b/examples/parallelism/README.md
@@ -1,4 +1,4 @@
-Here you'll find some examples that demonstrate how to use Hamilton's parallelism features.
+Here you'll find some examples that demonstrate how to use Apache Hamilton's parallelism features.
Note: serialization caveats still apply. For more information see the
[parallel documentation](https://hamilton.dagworks.io/en/latest/concepts/parallel-task/).
diff --git a/examples/parallelism/lazy_threadpool_execution/README.md b/examples/parallelism/lazy_threadpool_execution/README.md
index f82b70f3..e374a957 100644
--- a/examples/parallelism/lazy_threadpool_execution/README.md
+++ b/examples/parallelism/lazy_threadpool_execution/README.md
@@ -20,7 +20,7 @@ that the DAG can be parallelized, i.e. the left most nodes can be executed in pa
When you execute `run.py`, you will output that shows:
1. The DAG running in parallel -- check the image against what is printed.
-2. The DAG logging to the Hamilton UI -- please adjust for you project.
+2. The DAG logging to the Apache Hamilton UI -- please adjust for you project.
3. The DAG running without the adapter -- this is to show the difference in execution time.
4. An async version of the DAG running in parallel -- this is to show that the performance of this approach is similar.
diff --git a/examples/parallelism/star_counting/README.md b/examples/parallelism/star_counting/README.md
index 6b54ee84..f307238b 100644
--- a/examples/parallelism/star_counting/README.md
+++ b/examples/parallelism/star_counting/README.md
@@ -1,6 +1,6 @@
-# Star counting with Hamilton/Parallelism
+# Star counting with Apache Hamilton/Parallelism
-This example goes along with the "Counting Stars with Hamilton" blog post.
+This example goes along with the "Counting Stars with Apache Hamilton" blog post.
# Running
diff --git a/examples/people_data_labs/README.md b/examples/people_data_labs/README.md
index d43ac0e3..be032a66 100644
--- a/examples/people_data_labs/README.md
+++ b/examples/people_data_labs/README.md
@@ -2,13 +2,13 @@
[People Data Labs](https://www.peopledatalabs.com/) is a data provider that offers several data APIs for person enrichment & search, company enrichment & search, and IP enrichment.
-This example showcases how Hamilton can help you write modular data transformations.
+This example showcases how Apache Hamilton can help you write modular data transformations.
-> Dataflow visualization generated directly from the Hamilton code.
+> Dataflow visualization generated directly from the Apache Hamilton code.
## Content
-- `notebook.ipynb` is a step-by-step introduction to Hamilton. Start there.
+- `notebook.ipynb` is a step-by-step introduction to Apache Hamilton. Start there.
- `analysis.py` contains data transformations used by `run.py`. They're the same as the ones defined in `notebook.ipynb`.
- `run.py` contains code to execute the analysis.
@@ -42,6 +42,6 @@ You can even run this example in Google Colab:
## Resources
- [PDL Blog](https://blog.peopledatalabs.com/) and [PDL Recipes](https://docs.peopledatalabs.com/recipes)
-- [Interactive Hamilton training](https://www.tryhamilton.dev/hamilton-basics/jumping-in)
-- [Hamilton documentation](https://hamilton.dagworks.io/en/latest/concepts/node/)
-- more [Hamilton code examples](https://github.com/apache/hamilton/tree/main/examples) and integrations with Python tools.
+- [Interactive Apache Hamilton training](https://www.tryhamilton.dev/hamilton-basics/jumping-in)
+- [Apache Hamilton documentation](https://hamilton.dagworks.io/en/latest/concepts/node/)
+- more [Apache Hamilton code examples](https://github.com/apache/hamilton/tree/main/examples) and integrations with Python tools.
diff --git a/examples/plotly/README.md b/examples/plotly/README.md
index 99bf3004..941db33e 100644
--- a/examples/plotly/README.md
+++ b/examples/plotly/README.md
@@ -9,7 +9,7 @@ You need to install `plotly` (low-level API) to annotate your function with `plo
def confusion_matrix(...) -> plotly.graph_objects.Figure:
return plotly.express.imshow(...)
-# 2. import the module and create the Hamilton driver
+# 2. import the module and create the Apache Hamilton driver
dr = (
driver.Builder()
.with_config({...})
diff --git a/examples/polars/README.md b/examples/polars/README.md
index d6ba8176..45c9aaab 100644
--- a/examples/polars/README.md
+++ b/examples/polars/README.md
@@ -1,4 +1,4 @@
-# Classic Hamilton Hello World
+# Classic Apache Hamilton Hello World
In this example we show you how to create a simple hello world dataflow that
creates a polars dataframe as a result. It performs a series of transforms on the
@@ -9,7 +9,7 @@ File organization:
* `my_functions.py` houses the logic that we want to compute.
Note (1) how the functions are named, and what input
parameters they require. That is how we create a DAG modeling the dataflow we want to happen.
-* `my_script.py` houses how to get Hamilton to create the DAG, specifying that we want a polars dataframe and
+* `my_script.py` houses how to get Apache Hamilton to create the DAG, specifying that we want a polars dataframe and
exercise it with some inputs.
To run things:
@@ -31,13 +31,13 @@ Here is the graph of execution - which should look the same as the pandas exampl
# Caveat with Polars
There is one major caveat with Polars to be aware of: THERE IS NO INDEX IN POLARS LIKE THERE IS WITH PANDAS.
-What this means is that when you tell Hamilton to execute and return a polars dataframe if you are using the
+What this means is that when you tell Apache Hamilton to execute and return a polars dataframe if you are using the
[provided results builder](https://github.com/apache/hamilton/blob/sf-hamilton-1.14.1/hamilton/plugins/h_polars.py#L8), i.e. `hamilton.plugins.h_polars.PolarsResultsBuilder`, then you will have to
ensure the row order matches the order you expect for all the outputs you request. E.g. if you do a filter, or a sort,
-or a join, or a groupby, you will have to ensure that when you ask Hamilton to materialize an output that it's in the
+or a join, or a groupby, you will have to ensure that when you ask Apache Hamilton to materialize an output that it's in the
order you expect.
If you have questions, or need help with this example,
join us on [slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g), and we'll try to help!
-Otherwise if you have ideas on how to better make Hamilton work with Polars, please open an issue or start a discussion!
+Otherwise if you have ideas on how to better make Apache Hamilton work with Polars, please open an issue or start a discussion!
diff --git a/examples/polars/lazyframe/README.md b/examples/polars/lazyframe/README.md
index 90098503..23dbe5fa 100644
--- a/examples/polars/lazyframe/README.md
+++ b/examples/polars/lazyframe/README.md
@@ -1,4 +1,4 @@
-# Classic Hamilton Hello World
+# Classic Apache Hamilton Hello World
In this example we show you how to create a simple hello world dataflow that
creates a polars lazyframe as a result. It performs a series of transforms on the
@@ -9,7 +9,7 @@ File organization:
* `my_functions.py` houses the logic that we want to compute.
Note (1) how the functions are named, and what input
parameters they require. That is how we create a DAG modeling the dataflow we want to happen.
-* `my_script.py` houses how to get Hamilton to create the DAG, specifying that we want a polars dataframe and
+* `my_script.py` houses how to get Apache Hamilton to create the DAG, specifying that we want a polars dataframe and
exercise it with some inputs.
To run things:
@@ -25,13 +25,13 @@ Here is the graph of execution - which should look the same as the pandas exampl
# Caveat with Polars
There is one major caveat with Polars to be aware of: THERE IS NO INDEX IN POLARS LIKE THERE IS WITH PANDAS.
-What this means is that when you tell Hamilton to execute and return a polars dataframe if you are using the
+What this means is that when you tell Apache Hamilton to execute and return a polars dataframe if you are using the
[provided results builder](https://github.com/apache/hamilton/blob/sf-hamilton-1.14.1/hamilton/plugins/h_polars.py#L8), i.e. `hamilton.plugins.h_polars.PolarsResultsBuilder`, then you will have to
ensure the row order matches the order you expect for all the outputs you request. E.g. if you do a filter, or a sort,
-or a join, or a groupby, you will have to ensure that when you ask Hamilton to materialize an output that it's in the
+or a join, or a groupby, you will have to ensure that when you ask Apache Hamilton to materialize an output that it's in the
order you expect.
If you have questions, or need help with this example,
join us on [slack](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g), and we'll try to help!
-Otherwise if you have ideas on how to better make Hamilton work with Polars, please open an issue or start a discussion!
+Otherwise if you have ideas on how to better make Apache Hamilton work with Polars, please open an issue or start a discussion!
diff --git a/examples/prefect/README.md b/examples/prefect/README.md
index 4b632be4..42fad20b 100644
--- a/examples/prefect/README.md
+++ b/examples/prefect/README.md
@@ -1,13 +1,13 @@
-# Hamilton + Prefect
+# Apache Hamilton + Prefect
-In this example, were going to show how to run a simple `data preprocessing -> model training -> model evaluation` workflow using Hamilton within Prefect tasks.
+In this example, were going to show how to run a simple `data preprocessing -> model training -> model evaluation` workflow using Apache Hamilton within Prefect tasks.
- [**Prefect**](https://prefect.io) is an open source orchestrator written in Python. Its purpose is to launch and execute your workflows over your infrastructure. It is a macro-orchestration system.
-- [**Hamilton**](https://github.com/apache/hamilton) is a micro-framework to describe dataflows in Python. Its strength is expressing the flow of data & computation in a way that is straightforward to create, maintain, and reuse (much like DBT has done for SQL). It is lightweight and can be run anywhere that python runs. It is a micro-orchestration system.
+- [**Apache Hamilton**](https://github.com/apache/hamilton) is a micro-framework to describe dataflows in Python. Its strength is expressing the flow of data & computation in a way that is straightforward to create, maintain, and reuse (much like DBT has done for SQL). It is lightweight and can be run anywhere that python runs. It is a micro-orchestration system.
## Why both?
Prefect is a heavy weight system that includes useful features for production system such as job queues, caching, retries, logging, and more. However, each additional "task" you create comes with execution overhead and needs to be manually added to the DAG structure. Opting for less granular tasks ultimately reduces visibility and maintainability, while opting for more creates more overhead.
-In contrast, Hamilton automatically builds its DAG from the function definition itself, encouraging clean, small and single-purposed functions. By using Hamilton in a Prefect task, you gain this greater insight into lineage, reduced Prefect execution overhead and a simpler DAG to manage, and you get portable Python code you can run outside Prefect too.
+In contrast, Apache Hamilton automatically builds its DAG from the function definition itself, encouraging clean, small and single-purposed functions. By using Apache Hamilton in a Prefect task, you gain this greater insight into lineage, reduced Prefect execution overhead and a simpler DAG to manage, and you get portable Python code you can run outside Prefect too.
## What does the code do?
This example illustrates a typical data science / machine learning workflow where we:
@@ -17,11 +17,11 @@ This example illustrates a typical data science / machine learning workflow wher
4. Train a model
5. Evaluate the model to decide if it meets performance requirements
-The code to support these steps is written as modular functions in `prepare_data.py` (steps 1, 2, 3), `train_model.py` (step 4), and `evaluate_model.py` (step 5). The modules can be loaded with the Hamilton `Driver` to automatically generate the execution DAG from the functions. In our scenario, we use Prefect because we want to be able to schedule and run our workflow daily. The file `run.py` contains the Prefect workflow definition, where each task has its own Hamilton `Driver`. ([Lear [...]
+The code to support these steps is written as modular functions in `prepare_data.py` (steps 1, 2, 3), `train_model.py` (step 4), and `evaluate_model.py` (step 5). The modules can be loaded with the Apache Hamilton `Driver` to automatically generate the execution DAG from the functions. In our scenario, we use Prefect because we want to be able to schedule and run our workflow daily. The file `run.py` contains the Prefect workflow definition, where each task has its own Apache Hamilton `D [...]
You'll notice that the Prefect workflow has 2 tasks: `prepare_data_task` and `train_and_evaluate_model_task` even though there are 3 function modules. While each function module best describes a logical grouping of functions, the Prefect tasks can help define how/where they should be executed. For example, the data processing task might require CPU resources while the model training and evaluation would benefit from having access to a GPU. This split allows us to benefit from Prefect's m [...]
-For another workflow, a large Hamilton DAG in a single Prefect task, or smaller Prefect tasks and Hamilton modules might have been preferred. This shows how the Hamilton `Driver` brings flexibility to where and how you run your code.
+For another workflow, a large Apache Hamilton DAG in a single Prefect task, or smaller Prefect tasks and Apache Hamilton modules might have been preferred. This shows how the Apache Hamilton `Driver` brings flexibility to where and how you run your code.
## Prefect setup
The easiest way to get this example running is to sign up for Prefect's free tier and follow the [Prefect Cloud Quickstart](https://docs.prefect.io/latest/cloud/cloud-quickstart/) section. Don't want to sign up? Don't worry, as Prefect is open source, can instead opt to host and run everything yourself as described [here](https://docs.prefect.io/latest/host/). The steps to get started are:
@@ -43,11 +43,11 @@ The easiest way to get this example running is to sign up for Prefect's free tie
## Tips
-1. Use Prefect [Blocks](https://docs.prefect.io/latest/concepts/blocks/) to store your Hamilton configuration. This way, you can edit it directly from your Prefect dashboard to launch different runs without altering your source code.
+1. Use Prefect [Blocks](https://docs.prefect.io/latest/concepts/blocks/) to store your Apache Hamilton configuration. This way, you can edit it directly from your Prefect dashboard to launch different runs without altering your source code.
2. Prefect keeps track of your flow and tasks config, which is useful to reproduce runs.
-3. Store Hamilton [DAG visualization](https://hamilton.dagworks.io/en/latest/how-tos/use-hamilton-for-lineage/) with your runs using Prefect [artifacts](https://docs.prefect.io/latest/concepts/artifacts/).
+3. Store Apache Hamilton [DAG visualization](https://hamilton.dagworks.io/en/latest/how-tos/use-hamilton-for-lineage/) with your runs using Prefect [artifacts](https://docs.prefect.io/latest/concepts/artifacts/).
The general idea is:
1. produce a local visualization file
@@ -89,6 +89,6 @@ The easiest way to get this example running is to sign up for Prefect's free tie
create_link_artifact(
key=...,
link=f"{remote_basepath}/{remote_dag_file},
- description="Hamilton execution DAG"
+ description="Apache Hamilton execution DAG"
)
```
diff --git a/examples/ray/README.md b/examples/ray/README.md
index a7014e31..03cf752f 100644
--- a/examples/ray/README.md
+++ b/examples/ray/README.md
@@ -1,4 +1,4 @@
-# Scaling Hamilton on Ray
+# Scaling Apache Hamilton on Ray
[Ray](https://ray.io) is a general purpose framework that allows for parallel
computation on a local machine, as well as scaling to a
@@ -8,8 +8,8 @@ Ray cluster.
## injecting @ray.remote
See the `hello_world` example for how you might accomplish
-scaling Hamilton on Ray, where `@ray.remote` is injected around
-each Hamilton function.
+scaling Apache Hamilton on Ray, where `@ray.remote` is injected around
+each Apache Hamilton function.
## creating tasks
For [this paralle task approach](https://hamilton.dagworks.io/en/latest/concepts/parallel-task/) see [this
diff --git a/examples/ray/hello_world/README.md b/examples/ray/hello_world/README.md
index 9a412395..f2c40d10 100644
--- a/examples/ray/hello_world/README.md
+++ b/examples/ray/hello_world/README.md
@@ -1,7 +1,7 @@
-# Hamilton on Ray
+# Apache Hamilton on Ray
Here we have a hello world example showing how you can
-take some Hamilton functions and then easily run them
+take some Apache Hamilton functions and then easily run them
in a distributed setting via ray.
`pip install sf-hamilton[ray]` or `pip install sf-hamilton ray` to for the right dependencies to run this example.
diff --git a/examples/scikit-learn/species_distribution_modeling/README.md b/examples/scikit-learn/species_distribution_modeling/README.md
index 3734cb7a..0d084531 100644
--- a/examples/scikit-learn/species_distribution_modeling/README.md
+++ b/examples/scikit-learn/species_distribution_modeling/README.md
@@ -1,11 +1,11 @@
# Species distribution modeling
-We translate the Species distribution modeling from scikit-learn into Hamilton to showcase `pipe` and`pipe_output`.
+We translate the Species distribution modeling from scikit-learn into Apache Hamilton to showcase `pipe` and`pipe_output`.
# Highlights
Example of a simple ETL pipeline broken into modules with external couplings.
-1) To see how to couple external modules / source code and integrate it into a Hamilton DAG check out `grids.py` or `preprocessing.py`, where we use `@pipe` to wrap and inject external functions as Hamilton nodes.
+1) To see how to couple external modules / source code and integrate it into a Apache Hamilton DAG check out `grids.py` or `preprocessing.py`, where we use `@pipe` to wrap and inject external functions as Apache Hamilton nodes.
2) To see how to re-use functions check out `train_and_predict.py`, where we use `@pipe_output` to evaluate our model on the individual test and train datasets.
@@ -55,7 +55,7 @@ qux(baz(foo(bar)))
This can be particularly useful when:
1. Ensuring consistency between transforms/data processing steps across different nodes.
2. Re-using the same function on multiple nodes. For example, needing to do different pre-processing, but than passing data to the same `model`, aka feature engineering hyper-tuning.
-3. We can also use `step(...).when(...)` and can choose at execution time which transformation will be applied to the output of a particular node. For example, each `step` represents a different `model` and we switch between them with a config dictionary in the Hamilton driver.
+3. We can also use `step(...).when(...)` and can choose at execution time which transformation will be applied to the output of a particular node. For example, each `step` represents a different `model` and we switch between them with a config dictionary in the Apache Hamilton driver.
## original_script.py
The original script has been left intact. The only change we made is to add
@@ -84,7 +84,7 @@ python -m run.py
# Disclaimer
The original code and analysis is taken from [scikit-learn](https://scikit-learn.org/stable/auto_examples/applications/plot_species_distribution_modeling.html).
-We don't know them and they don't know us - but they wrote a neat analysis and we wanted to show you how their procedural code would look like if it was written with the help of Hamilton.
+We don't know them and they don't know us - but they wrote a neat analysis and we wanted to show you how their procedural code would look like if it was written with the help of Apache Hamilton.
Thanks to the authors for creating it:
- Peter Prettenhofer <pe...@gmail.com>,
diff --git a/examples/scikit-learn/transformer/README.md b/examples/scikit-learn/transformer/README.md
index 1b2d5441..1fe1eb69 100644
--- a/examples/scikit-learn/transformer/README.md
+++ b/examples/scikit-learn/transformer/README.md
@@ -1,4 +1,4 @@
-# Classic Hamilton Hello World
+# Classic Apache Hamilton Hello World
In this example we show you a custom scikit-learn `Transformer` class. This class should be compliant with [scikit-learn transformers specifications](https://scikit-learn.org/stable/developers/develop.html). This class is meant to be used as part of broader scikit-learn pipelines. Scikit-learn estimators and pipelines allow for stateful objects, which are helpful when applying transformations on train-test splits notably. Also, all pipeline, estimator, and transformer objects should be p [...]
@@ -23,7 +23,7 @@ Here is the visualization of the execution that the transformer currently perfor
# Limitations and TODOs
-- The current implementation relies on Hamilton defaults' `base.HamiltonGraphAdapter` and `base.PandasDataFrameResult` which limits the compatibility with other computation engines supported by Hamilton
-- The current implementation could be improved for deeper object inspection. A particular challenge is that the Hamilton driver alters the number of columns / features of the input array, and does so by specifying the *output columns*. In contrast, scikit-learn typically reads columns / features of the *input array* and passes it down. It would be worth looking at [the output feature naming convention](https://scikit-learn.org/stable/developers/develop.html#developer-api-for-set-output) [...]
-- The current implementation allows little direct access to the `Hamilton driver`. Currently, the driver accessible via the `.driver_` attribute after calling `.fit()` or `.fit_transform()` which seems to be coherent with typical scikit-learn behavior.
+- The current implementation relies on Apache Hamilton defaults' `base.HamiltonGraphAdapter` and `base.PandasDataFrameResult` which limits the compatibility with other computation engines supported by Apache Hamilton
+- The current implementation could be improved for deeper object inspection. A particular challenge is that the Apache Hamilton driver alters the number of columns / features of the input array, and does so by specifying the *output columns*. In contrast, scikit-learn typically reads columns / features of the *input array* and passes it down. It would be worth looking at [the output feature naming convention](https://scikit-learn.org/stable/developers/develop.html#developer-api-for-set-o [...]
+- The current implementation allows little direct access to the `Apache Hamilton driver`. Currently, the driver accessible via the `.driver_` attribute after calling `.fit()` or `.fit_transform()` which seems to be coherent with typical scikit-learn behavior.
- The current implementation could be slightly modified to have no dependencies on scikit-learn itself. However, having inheriting from `BaseEstimator` and `TransformerMixin` provides some clarity about the class's purpose.
diff --git a/examples/snowflake/hamilton_ui/README.md b/examples/snowflake/hamilton_ui/README.md
index 9f5fc717..05d58a9f 100644
--- a/examples/snowflake/hamilton_ui/README.md
+++ b/examples/snowflake/hamilton_ui/README.md
@@ -1,16 +1,16 @@
-# Running the Hamilton & the Hamilton UI in Snowflake
+# Running the Apache Hamilton & the Apache Hamilton UI in Snowflake
-This example is code for the ["Observability of Python code and application logic with Hamilton UI on Snowflake Container Services" post](https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635) by
+This example is code for the ["Observability of Python code and application logic with Apache Hamilton UI on Snowflake Container Services" post](https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635) by
[Greg Kantyka](https://medium.com/@pkantyka).
Here we show the code required to be packaged up for use on Snowflake:
-1. Docker file that runs the Hamilton UI and a flask endpoint to exercise Hamilton code
-2. my_functions.py - the Hamilton code that is exercised by the flask endpoint
-3. pipeline_endpoint.py - the flask endpoint that exercises the Hamilton code
+1. Docker file that runs the Apache Hamilton UI and a flask endpoint to exercise Apache Hamilton code
+2. my_functions.py - the Apache Hamilton code that is exercised by the flask endpoint
+3. pipeline_endpoint.py - the flask endpoint that exercises the Apache Hamilton code
To run see:
- snowflake.sql that contains all the SQL to create the necessary objects in Snowflake and exercise things.
-For more details see ["Observability of Python code and application logic with Hamilton UI on Snowflake Container Services" post](https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635) by
+For more details see ["Observability of Python code and application logic with Apache Hamilton UI on Snowflake Container Services" post](https://medium.com/@pkantyka/observability-of-python-code-and-application-logic-with-hamilton-ui-on-snowflake-container-services-a26693b46635) by
[Greg Kantyka](https://medium.com/@pkantyka).
diff --git a/examples/spark/README.md b/examples/spark/README.md
index e06a2ccc..d6eb61bc 100644
--- a/examples/spark/README.md
+++ b/examples/spark/README.md
@@ -1,19 +1,19 @@
-# Scaling Hamilton on Spark
+# Scaling Apache Hamilton on Spark
## Pyspark
-If you're using pyspark, Hamilton allows for natural manipulation of pyspark dataframes,
+If you're using pyspark, Apache Hamilton allows for natural manipulation of pyspark dataframes,
with some special constructs for managing DAGs of UDFs.
See the examples in `pyspark` & `world_of_warcraft` and `tpc-h` to learn more.
## Pandas
-If you're using Pandas, Hamilton scales by using Koalas on Spark.
+If you're using Pandas, Apache Hamilton scales by using Koalas on Spark.
Koalas became part of Spark officially in Spark 3.2, and was renamed Pandas on Spark.
The example in `pandas_on_spark` here assumes that.
## Pyspark UDFs
-If you're not using Pandas, then you can use Hamilton to manage and organize your pyspark UDFs.
+If you're not using Pandas, then you can use Apache Hamilton to manage and organize your pyspark UDFs.
See the example in `pyspark_udfs`.
Note: we're looking to expand coverage and support for more Spark use cases. Please come find us, or open an issue,
@@ -21,7 +21,7 @@ if you have a use case that you'd like to see supported!
## Caveats
-Hamilton's type-checking doesn't inherently work with spark connect, which can swap out different types of dataframes that
+Apache Hamilton's type-checking doesn't inherently work with spark connect, which can swap out different types of dataframes that
are not part of the same subclass. Thus when you are passing in a spark session or spark dataframe as an input,
you can use the `SparkInputValidator` (available by instantiation or access of the static field `h_spark.SPARK_INPUT_CHECK`).
diff --git a/examples/spark/pandas_on_spark/README.md b/examples/spark/pandas_on_spark/README.md
index 5b32f5fc..3c93c5cd 100644
--- a/examples/spark/pandas_on_spark/README.md
+++ b/examples/spark/pandas_on_spark/README.md
@@ -1,7 +1,7 @@
-# Hamilton on Koalas Spark 3.2+
+# Apache Hamilton on Koalas Spark 3.2+
Here we have a hello world example showing how you can
-take some Hamilton functions and then easily run them
+take some Apache Hamilton functions and then easily run them
in a distributed setting via Spark 3.2+ using Koalas.
`pip install sf-hamilton[pyspark]` or `pip install sf-hamilton pyspark[pandas_on_spark]` to for the right dependencies to run this example.
diff --git a/examples/spark/pyspark/README.md b/examples/spark/pyspark/README.md
index b0527a16..feb01663 100644
--- a/examples/spark/pyspark/README.md
+++ b/examples/spark/pyspark/README.md
@@ -1,15 +1,15 @@
-# Hamilton and Pyspark
+# Apache Hamilton and Pyspark
-**TL;DR** Hamilton now has a full pyspark integration. This enables you to build modular pyspark applications
+**TL;DR** Apache Hamilton now has a full pyspark integration. This enables you to build modular pyspark applications
by declaring each transformation as a hamilton function. The new `with_columns` decorator enables you to specify a
series of individual column operations to a dataframe, written as pandas UDFs, python UDFs, or function of pyspark dataframes.
-Together with vanilla Hamilton, this new integration can help you break complex pyspark code into a series of self-contained, unit-testable functions.
+Together with vanilla Apache Hamilton, this new integration can help you break complex pyspark code into a series of self-contained, unit-testable functions.
<p align="center">
<img width="600" height="600" src="grouped_transformations.png">
</p>
-*A spark application written with Hamilton that represents multiple joins (in blue), a set of map operations (in green) and a set of join/filters (in yellow). This uses Hamiltonβs visualization features (with a little extra annotation). See [TPC-H query 8](../tpc-h/query_8.py) for motivation.*
+*A spark application written with Apache Hamilton that represents multiple joins (in blue), a set of map operations (in green) and a set of join/filters (in yellow). This uses Apache Hamiltonβs visualization features (with a little extra annotation). See [TPC-H query 8](../tpc-h/query_8.py) for motivation.*
## Apache Spark
[Apache Spark](https://spark.apache.org/) (and its python API, [pyspark](https://spark.apache.org/docs/latest/api/python/index.html)) is an open-source library for building out highly scalable data transformations.
@@ -33,20 +33,20 @@ Specifically, we've observed the following problems with pyspark code:
5. _They are notoriously tricky to debug._ Large pipelines of spark transformations (much like SQL transformations) will often have errors that cascade upwards, and pinpointing the source of these can be quite a challenge.
-# Hamilton
-As this is a README inside the Hamilton repository, we assume some basic familiarity. That said, here's a quick primer:
+# Apache Hamilton
+As this is a README inside the Apache Hamilton repository, we assume some basic familiarity. That said, here's a quick primer:
-Hamilton is an open-source Python framework for writing data transformations.
-One writes Python functions in a declarative style, which Hamilton parses into nodes in
+Apache Hamilton is an open-source Python framework for writing data transformations.
+One writes Python functions in a declarative style, which Apache Hamilton parses into nodes in
a graph based on their names, arguments and type annotations. The simple rule is akin to that of pytest fixtures --
the name of a parameter points to another node (function) in the graph, and the name of the function defines a referencable node.
-You can request specific outputs, and Hamilton will execute the required nodes (specified by your functions) to produce them.
+You can request specific outputs, and Apache Hamilton will execute the required nodes (specified by your functions) to produce them.
You can try hamilton out in your browser at [tryhamilton.dev](https://tryhamilton.dev).
# Integration
-Breaking your spark application into Hamilton functions with pyspark dataframes as inputs and outputs gets you most of
+Breaking your spark application into Apache Hamilton functions with pyspark dataframes as inputs and outputs gets you most of
the way towards more modular/documented code.
That said, it falls flat in a critical area β column-level lineage/transformation
simplicity. For complex series of map operations, spark represents all transformations
@@ -58,7 +58,7 @@ Spark, however, has no notion of indices. Data is partitioned across a cluster,
Thus, the two options one previously had for integrating with pyspark both have disadvantages:
1. Extracting into columns then joining is prohibitively expensive and taxing on the spark optimizer (which we have not found was smart enough to detect this pattern)
-2. Running pure DataFrame transformations does not afford the expressiveness that Hamilton provides.
+2. Running pure DataFrame transformations does not afford the expressiveness that Apache Hamilton provides.
The idea is to break your transformations into sections that form one of two shapes.
@@ -66,7 +66,7 @@ The idea is to break your transformations into sections that form one of two sha
2. Form a DAG of column-level operations (for cardinality-preserving operations)
For the first case, we just use the pyspark dataframe API. You define functions that, when put
-through Hamilton, act as a pipe. For example:
+through Apache Hamilton, act as a pipe. For example:
### Joins/Aggregations/Filters
@@ -296,10 +296,10 @@ normal set of spark operations.
## Scaling Alternatives
-Pyspark is not the only way to scale up your computation. Hamilton supports `pandas-on-spark` as well. You can use pandas-on-spark with the `KoalaGraphAdapter` -- see [Pandas on Spark](../pandas_on_spark/README.md) for reference.
-Some people prefer vanilla spark, some like pandas-on-spark. We support both. Hamilton also support executing map-based pandas UDFs in pyspark, in case you want simple parallelism. See [pyspark_udfs](../pyspark_udfs/README.md) for reference.
+Pyspark is not the only way to scale up your computation. Apache Hamilton supports `pandas-on-spark` as well. You can use pandas-on-spark with the `KoalaGraphAdapter` -- see [Pandas on Spark](../pandas_on_spark/README.md) for reference.
+Some people prefer vanilla spark, some like pandas-on-spark. We support both. Apache Hamilton also support executing map-based pandas UDFs in pyspark, in case you want simple parallelism. See [pyspark_udfs](../pyspark_udfs/README.md) for reference.
-Hamilton has integrations with other scaling libraries as well -- it all depends on your use-case:
+Apache Hamilton has integrations with other scaling libraries as well -- it all depends on your use-case:
- [dask](../../dask/README.md)
- [ray](../../ray/README.md)
diff --git a/examples/spark/pyspark_udfs/README.md b/examples/spark/pyspark_udfs/README.md
index 0db748bc..c24e1fd1 100644
--- a/examples/spark/pyspark_udfs/README.md
+++ b/examples/spark/pyspark_udfs/README.md
@@ -1,6 +1,6 @@
-# Using Hamilton to help manage your pyspark UDFs
+# Using Apache Hamilton to help manage your pyspark UDFs
-Here we have a version of the Hamilton hello world example showing how you can take some Hamilton functions and then
+Here we have a version of the Apache Hamilton hello world example showing how you can take some Apache Hamilton functions and then
easily run them as pyspark UDFs.
## Dependencies
@@ -9,8 +9,8 @@ easily run them as pyspark UDFs.
## File organization:
-* `pandas_udfs.py` contains UDF definitions that Hamilton will know how to run as `pandas_udfs` on pyspark.
-* `vanilla_udfs.py` contains UDF definitions that Hamilton will know how to run as row based UDFs on pyspark.
+* `pandas_udfs.py` contains UDF definitions that Apache Hamilton will know how to run as `pandas_udfs` on pyspark.
+* `vanilla_udfs.py` contains UDF definitions that Apache Hamilton will know how to run as row based UDFs on pyspark.
* `run.py` contains spark driver and hamilton driver code to run everything.
## To run
@@ -33,35 +33,35 @@ You should get the following as output:
| 50| 400|43.333333333333336| 0.125| 1.2579336|1.0|2.0| 3.0|
+-----+-------+------------------+----------------+-----------------------------+---+---+--------------+
```
-with Hamilton showing you the following execution visualization:
+with Apache Hamilton showing you the following execution visualization:
Note: foo, bar, and augmented_mean do not depend on any inputs in the dataframe.
-# What Hamilton does under the hood
+# What Apache Hamilton does under the hood
-You need to use the `PySparkUDFGraphAdapter` for this to work. This is because Hamilton needs to know how to
-augment operations for pyspark. With this adapter, Hamilton will:
+You need to use the `PySparkUDFGraphAdapter` for this to work. This is because Apache Hamilton needs to know how to
+augment operations for pyspark. With this adapter, Apache Hamilton will:
1. Takes care of the spark boilerplate for you in creating a UDF.
2. Intelligently knows whether to pull from the dataframe for input to a function, based on column names,
or pulls from the passed in dictionary of inputs for that UDF.
3. **Append** the requested outputs to the dataframe.
4. Operate on the premise that as part of the inputs field to `dr.execute(..., inputs=INPUTS)`, the `INPUTS`
-dictionary contains a mapping of column name to pyspark dataframe. Hamilton will then know to use that **single**
+dictionary contains a mapping of column name to pyspark dataframe. Apache Hamilton will then know to use that **single**
dataframe to pull from and append columns to, to coordinate UDF execution.
5. Assume all functions are `map` based functions. If they're not, then you'll need to manually
-manage that outside of Hamilton.
+manage that outside of Apache Hamilton.
-# Why is Hamilton useful here?
+# Why is Apache Hamilton useful here?
1. It makes it easy to write UDFs that can be run in a distributed setting.
2. Those UDFs can be run in other contexts that don't use spark!
-3. Hamilton forces you to curate UDFs and thus they subsequently can be versioned, tested, documented, and reused
+3. Apache Hamilton forces you to curate UDFs and thus they subsequently can be versioned, tested, documented, and reused
in other spark jobs easily. Think reusable feature definitions, think reusable metric definitions! Assumption: they
have to be `map` based functions.
4. Anyone who wants to use these UDFs, MUST ALIGN ON COLUMNS NAMES! This is a good thing, because it means that you can
standardize on column names and reuse UDFs across different spark jobs, which lowers your maintenance burden!
-5. If you don't care about intermediate outputs, Hamilton won't add them. It'll only add the outputs you want to the
+5. If you don't care about intermediate outputs, Apache Hamilton won't add them. It'll only add the outputs you want to the
passed in dataframe.
# Limitations
@@ -81,7 +81,7 @@ Let us know if this would be useful to you!
## All the Pandas UDF signatures
(1) Let us know what you need.
-(2) Implementation is a matter of (a) getting the API right, and (b) making sure it fits with the Hamilton way of thinking.
+(2) Implementation is a matter of (a) getting the API right, and (b) making sure it fits with the Apache Hamilton way of thinking.
## Aggregation UDFs
@@ -94,8 +94,8 @@ a good API for this.
# Other questions
-## Can't I just use pyspark dataframes directly with Hamilton functions?
+## Can't I just use pyspark dataframes directly with Apache Hamilton functions?
-Yes, with Hamilton you could write functions that define a named flow that operates entirely over pyspark dataframes.
-However, you lose a lot of the flexibility of Hamilton doing things that way. We're open to suggestions,
+Yes, with Apache Hamilton you could write functions that define a named flow that operates entirely over pyspark dataframes.
+However, you lose a lot of the flexibility of Apache Hamilton doing things that way. We're open to suggestions,
or API changes to make this approach easier.
diff --git a/examples/streamlit/README.md b/examples/streamlit/README.md
index 1b261304..9f99de66 100644
--- a/examples/streamlit/README.md
+++ b/examples/streamlit/README.md
@@ -1,4 +1,4 @@
-# Streamlit + Hamilton
+# Streamlit + Apache Hamilton
> This example accompanies the documentation page for [Streamlit](https://hamilton.dagworks.io/en/latest/integrations/streamlit/) integration.
@@ -14,23 +14,23 @@ In this example, We will build a simple financial dashboard based on the Kaggle
## File organization
-Adding Hamilton to your Streamlit application can provide a better separation between the dataflow and the UI logic. They pair nicely together because Hamilton is also stateless. Once defined, each call to `Driver.execute()` is independent. Therefore, on each Streamlit rerun, you use `Driver.execute()` to complete computations. Using Hamilton this way allows you to write your dataflow into Python modules and outside of the Streamlit.
+Adding Apache Hamilton to your Streamlit application can provide a better separation between the dataflow and the UI logic. They pair nicely together because Apache Hamilton is also stateless. Once defined, each call to `Driver.execute()` is independent. Therefore, on each Streamlit rerun, you use `Driver.execute()` to complete computations. Using Apache Hamilton this way allows you to write your dataflow into Python modules and outside of the Streamlit.
### logic.py
-Hamilton transformations are defined in the module `logic.py`. This includes downloading the data from the web, getting unique values for `job`, conducting groupby aggregates, and creating `plotly` figures.
+Apache Hamilton transformations are defined in the module `logic.py`. This includes downloading the data from the web, getting unique values for `job`, conducting groupby aggregates, and creating `plotly` figures.
### app.py
The Streamlit UI is defined in `app.py`. Notice a few things:
- `app.py` doesn't have to depend on `pandas` and `plotly`.
- `@cache_resource` allows to create the `Driver` only once.
-- `@cache_data` on `_execute()` will automatically cache any Hamilton result based on the combination of arguments (`final_vars`, `inputs`, and `overrides`)
+- `@cache_data` on `_execute()` will automatically cache any Apache Hamilton result based on the combination of arguments (`final_vars`, `inputs`, and `overrides`)
- `get_state_inputs()` and `get_state_overrides()` will collect values from user inputs.
- `execute()` parses the inputs and overrides from the state and call `_execute()`.
## Benefits
- **Clearer scope**: the decoupling between `app.py` and `logic.py` makes it easier to add data transformations or extend UI, and debug errors associated with either.
-- **Reusable code**: the module `logic.py` can be reused elsewhere with Hamilton.
- - If you are building a proof-of-concept with Streamlit, your Hamilton module will be able to grow with your project and be useful for your production pipelines.
- - If you are already building dataflows with Hamilton, using it with Streamlit ensures your dashboard metrics have the same implementation with your production pipeline (i.e., prevent [implementation skew](https://building.nubank.com.br/dealing-with-train-serve-skew-in-real-time-ml-models-a-short-guide/))
-- **Performance boost**: by caching the Hamilton Driver and its execution call, we are able to effectively cache all data operations in a few lines of code. Furthermore, Hamilton can scale further by using a remote task executor on a separate machine from the Streamlit application.
+- **Reusable code**: the module `logic.py` can be reused elsewhere with Apache Hamilton.
+ - If you are building a proof-of-concept with Streamlit, your Apache Hamilton module will be able to grow with your project and be useful for your production pipelines.
+ - If you are already building dataflows with Apache Hamilton, using it with Streamlit ensures your dashboard metrics have the same implementation with your production pipeline (i.e., prevent [implementation skew](https://building.nubank.com.br/dealing-with-train-serve-skew-in-real-time-ml-models-a-short-guide/))
+- **Performance boost**: by caching the Apache Hamilton Driver and its execution call, we are able to effectively cache all data operations in a few lines of code. Furthermore, Apache Hamilton can scale further by using a remote task executor on a separate machine from the Streamlit application.
diff --git a/examples/styling_visualization/README.md b/examples/styling_visualization/README.md
index 3374d54a..5f88a466 100644
--- a/examples/styling_visualization/README.md
+++ b/examples/styling_visualization/README.md
@@ -1,6 +1,6 @@
# Style your dataflow visualizations
-A core feature of Hamilton is the ability to generate visualizations directly from your code. By default, each nodes is labelled with its name and type and is stylized (shape, color, outline, etc.).
+A core feature of Apache Hamilton is the ability to generate visualizations directly from your code. By default, each nodes is labelled with its name and type and is stylized (shape, color, outline, etc.).
Now, it is possible to customize the visualization style based on node attributes. To do so, you need to define a function that will be applied to each node of the graph.
@@ -13,7 +13,7 @@ def custom_style(
) -> Tuple[dict, Optional[str], Optional[str]]:
"""Custom style function for the visualization.
- :param node: node that Hamilton is styling.
+ :param node: node that Apache Hamilton is styling.
:param node_class: class used to style the default visualization
:return: a triple of (style, node_class, legend_name) where
style: dictionary of graphviz attributes https://graphviz.org/docs/nodes/,
@@ -51,7 +51,7 @@ This [online graphviz editor](https://edotor.net/) can help you get started!
## Use cases
-The Hamilton visualizations help you document your code and communicate what your dataflow does. Adapting the style to your given context can help highlight
+The Apache Hamilton visualizations help you document your code and communicate what your dataflow does. Adapting the style to your given context can help highlight
- code ownership
- data sources
- personally identifiable information
@@ -62,7 +62,7 @@ The Hamilton visualizations help you document your code and communicate what you
Having access to `graph_types.HamiltonNode` as input means you can customize based on node name, type, tags, originating function, and more, allowing for a great degree of flexibility.
## Node classes
-The default Hamilton visualization currently relies on 3 node classes:
+The default Apache Hamilton visualization currently relies on 3 node classes:
- `"function"`: default node type
- `"input"`: input node, i.e., that doesn't have an associated function
- `"config"`: value passed to the Driver config
diff --git a/examples/vaex/README.md b/examples/vaex/README.md
index beccd59e..9af30a4b 100644
--- a/examples/vaex/README.md
+++ b/examples/vaex/README.md
@@ -1,6 +1,6 @@
# Vaex Example
-This is a simple example taking the Hamilton hello world and converting it to run with Vaex.
+This is a simple example taking the Apache Hamilton hello world and converting it to run with Vaex.
To run:
diff --git a/examples/validation/static_validator/README.md b/examples/validation/static_validator/README.md
index 61938729..7af314e3 100644
--- a/examples/validation/static_validator/README.md
+++ b/examples/validation/static_validator/README.md
@@ -17,7 +17,7 @@ To run the example, run `python run.py` and you should see the following output:
# bad_module.py didn't have the right tags so graph building errored out
...
-Hamilton.lifecycle.base.ValidationException: Node validation failed! 1 errors encountered:
+Apache Hamilton.lifecycle.base.ValidationException: Node validation failed! 1 errors encountered:
foo: Node bad_module.foo is an output node, but does not have a table_name tag.
```
diff --git a/hamilton/plugins/README.md b/hamilton/plugins/README.md
index 9af5df6b..c6f4b821 100644
--- a/hamilton/plugins/README.md
+++ b/hamilton/plugins/README.md
@@ -1,6 +1,6 @@
# Plugins
-Hamilton enables plugins -- the requirement is that the core library according to the plugin is installed, and the plugin will be registered automatically.
+Apache Hamilton enables plugins -- the requirement is that the core library according to the plugin is installed, and the plugin will be registered automatically.
It is up to the user to install the plugin via the target, E.G. `hamilton[pyspark]`, which will install the correct depdendencies.
diff --git a/plugin_tests/README.md b/plugin_tests/README.md
index 6dfdab1f..00e4b53d 100644
--- a/plugin_tests/README.md
+++ b/plugin_tests/README.md
@@ -1,5 +1,5 @@
# What are these tests?
-These are "integration style" tests, that test the more heavy-weight plugin functionality Hamilton has. E.g. with GraphAdapters. So find/place tests here that exercise `h_*.py` modules.
+These are "integration style" tests, that test the more heavy-weight plugin functionality Apache Hamilton has. E.g. with GraphAdapters. So find/place tests here that exercise `h_*.py` modules.
For unit tests, place them under `tests/plugins/`.
diff --git a/ui/README.md b/ui/README.md
index ef774363..07aaad89 100644
--- a/ui/README.md
+++ b/ui/README.md
@@ -1,20 +1,20 @@
-# Hamilton UI
+# Apache Hamilton UI
-This contains the code for the new Hamilton UI. For an overview of getting started & features
+This contains the code for the new Apache Hamilton UI. For an overview of getting started & features
[see this documentation](https://hamilton.dagworks.io/en/latest/concepts/ui). For a lengthier post and intro see our [blog post](https://blog.dagworks.io/p/hamilton-ui-streamlining-metadata).
## One operational UI for all your dataflows
-The Hamilton UI is a system that provides the following capabilities:
+The Apache Hamilton UI is a system that provides the following capabilities:
1. Execution tracking with associated metadata
* Provides a persistent database to store/manage these
* Provides a server that allows reading/writing/authentication
-2. Data/artifact observability: provides telemetry/observability of Hamilton executions + specific function results/code through a web interface
+2. Data/artifact observability: provides telemetry/observability of Apache Hamilton executions + specific function results/code through a web interface
3. Lineage & provenance: allows you to quickly inspect how code and data is connected.
4. Catalog: everything is observed and cataloged, so you can quickly search and find what exists and when it was run.
-The UI is meant to monitor/debug Hamilton dataflows both in **development** and **production**. The aim is to enable
+The UI is meant to monitor/debug Apache Hamilton dataflows both in **development** and **production**. The aim is to enable
dataflow authors to move faster during all phases of the software development lifecycle.
For an overview of some of these features [you can watch this quick video](https://youtu.be/0VIVSeN7Ij8?si=maeV0zdzTPSqUl1N).
@@ -75,7 +75,7 @@ cd hamilton/ui
./run.sh
```
Once docker is running navigate to http://localhost:8242 and create an email and a project; then follow
-instructions on integrating with Hamilton.
+instructions on integrating with Apache Hamilton.
A fuller guide can be found [here](https://hamilton.dagworks.io/en/latest/concepts/ui).
diff --git a/ui/frontend/README.md b/ui/frontend/README.md
index c6d96405..fe8f1991 100644
--- a/ui/frontend/README.md
+++ b/ui/frontend/README.md
@@ -1,4 +1,4 @@
-# Hamilton UI frontend
+# Apache Hamilton UI frontend
## Dev setup
diff --git a/ui/sdk/README.md b/ui/sdk/README.md
index 4e306f3c..e14900e7 100644
--- a/ui/sdk/README.md
+++ b/ui/sdk/README.md
@@ -1,20 +1,20 @@
-# Hamilton UI SDK: Client Code & Related
+# Apache Hamilton UI SDK: Client Code & Related
-Welcome to using the Hamilton UI!
+Welcome to using the Apache Hamilton UI!
-Here are instructions on how to get started with tracking, and managing your Hamilton
-DAGs with the Hamilton UI.
+Here are instructions on how to get started with tracking, and managing your Apache Hamilton
+DAGs with the Apache Hamilton UI.
## Getting Started
For the latest documentation, please consult our
-[Hamilton documentation](https://hamilton.dagworks.io/) under `Hamilton UI`.
+[Apache Hamilton documentation](https://hamilton.dagworks.io/) under `Apache Hamilton UI`.
-For a quick overview of Hamilton, we suggest [tryhamilton.dev](https://www.tryhamilton.dev/).
+For a quick overview of Apache Hamilton, we suggest [tryhamilton.dev](https://www.tryhamilton.dev/).
## Using the HamiltonTracker
-First, you'll need to install the Hamilton SDK package. Assuming you're using pip, you
+First, you'll need to install the Apache Hamilton SDK package. Assuming you're using pip, you
can do this with:
```bash
@@ -25,7 +25,7 @@ pip install "sf-hamilton[sdk]"
python -c "from hamilton_sdk import adapters"
```
-Next, you'll need to modify your Hamilton driver. You'll only need to use one line of code to
+Next, you'll need to modify your Apache Hamilton driver. You'll only need to use one line of code to
replace your driver with ours:
```python
@@ -49,7 +49,7 @@ dr = (
```
*Project ID*: You'll need a project ID. Create a project if you don't have one, and take the ID from that.
-*username*: This is the email address you used to set up the Hamilton UI.
+*username*: This is the email address you used to set up the Apache Hamilton UI.
*dag_name*: for a project, the DAG name is the top level way to group DAGs.
E.g. ltv_model, us_sales, etc.
@@ -57,8 +57,8 @@ E.g. ltv_model, us_sales, etc.
*tags*: these are optional are string key value paris. They allow you to filter and curate
various DAG runs.
-Then run Hamilton as normal! Each DAG run will be tracked, and you'll have access to it in the
-Hamilton UI. After spinning up the Hamilton UI application, visit it to see your projects & DAGs.
+Then run Apache Hamilton as normal! Each DAG run will be tracked, and you'll have access to it in the
+Apache Hamilton UI. After spinning up the Apache Hamilton UI application, visit it to see your projects & DAGs.
# License
diff --git a/writeups/garbage_collection/post.md b/writeups/garbage_collection/post.md
index fb37e689..d061eeff 100644
--- a/writeups/garbage_collection/post.md
+++ b/writeups/garbage_collection/post.md
@@ -1,4 +1,4 @@
-# On Garbage Collection and Memory Optimization in Hamilton
+# On Garbage Collection and Memory Optimization in Apache Hamilton
## A chance to nerd out
@@ -6,13 +6,13 @@ A few weeks ago, one of our users came to us with an issue. They were processing
and noticed that the memory profile would quickly spiral out of control. The memory usage of the program grew until (a) it got prohibitively slow, then died, or (b) it just died due to a memory error.
This was, of course, because we had not optimized the memory at all. If I recall correctly, all we had was a few `TODO -- optimize memory usage` comments scattered throughout.
-Now that I had an excuse (a user asking me), I jumped into the rabbit hole of profiling, garbage collection, and optimization, with the goal of reducing Hamilton's memory overhead as much as possible.
+Now that I had an excuse (a user asking me), I jumped into the rabbit hole of profiling, garbage collection, and optimization, with the goal of reducing Apache Hamilton's memory overhead as much as possible.
-In this writeup I will talk a little about python garbage collection, share the change we made to improve Hamilton memory usage, and show its impact through a reproducible example.
+In this writeup I will talk a little about python garbage collection, share the change we made to improve Apache Hamilton memory usage, and show its impact through a reproducible example.
-If you're not familiar with Hamilton, you can get started at [tryhamilton.dev](www.tryhamilton.dev), or go to the [root](https://www.github.com/apache/hamilton) of this repository and read the README.
-If you don't want to read it and instead want to jump straight ahead, all you need to know is that Hamilton enables you to define your code as a directed acyclic graph (DAG) of python functions,
-and runs them for you. It differs from orchestration frameworks (Airflow, Metaflow, etc...) in that it is a micro orchestrator -- Hamilton does not provision compute or track executions. Rather, it is run as a library
+If you're not familiar with Apache Hamilton, you can get started at [tryhamilton.dev](www.tryhamilton.dev), or go to the [root](https://www.github.com/apache/hamilton) of this repository and read the README.
+If you don't want to read it and instead want to jump straight ahead, all you need to know is that Apache Hamilton enables you to define your code as a directed acyclic graph (DAG) of python functions,
+and runs them for you. It differs from orchestration frameworks (Airflow, Metaflow, etc...) in that it is a micro orchestrator -- Apache Hamilton does not provision compute or track executions. Rather, it is run as a library
on top of other more infrastructure-focused systems. Its primary aim is to help you organize your code and run on multiple platforms.
## Garbage collection: a primer
@@ -116,7 +116,7 @@ swap space, takes longer to handle memory retrieval/commitment, and eventually j
Now that we have a grasp on the problem and a profiling methodology, let's dig in to see what we can fix. Luckily, the root of the issue was fairly clear.
-To execute a Hamilton node and just its required upstream dependencies, we conduct a depth-first-traversal of the graph, storing results
+To execute a Apache Hamilton node and just its required upstream dependencies, we conduct a depth-first-traversal of the graph, storing results
we've realized in a `computed` dictionary. We use this for multiple purposes -- it can help us (a) avoid recomputing nodes and instead store
the results, and (b) return the final results at the end that we need. The problem is that we held onto all results, regardless of whether we would need them later.
In the (albeit contrived) script above, we only need the prior node in the chain to compute the current one.
@@ -169,7 +169,7 @@ You can see a similar pattern if you up the allocation size to 1gb. It occasiona
## Wrapping it up
-After solving this memory leak, we also fixed one more -- in Hamilton's [dynamic execution capabilities](https://blog.dagworks.io/p/counting-stars-with-hamilton),
+After solving this memory leak, we also fixed one more -- in Apache Hamilton's [dynamic execution capabilities](https://blog.dagworks.io/p/counting-stars-with-hamilton),
it was doing something similar for parallel execution -- storing all the results and not letting go of them.
You can find the original fix + some initial thoughts on the script/evidence as we learned this in the [PR](https://github.com/apache/hamilton/pull/374).
@@ -177,5 +177,5 @@ You can find the original fix + some initial thoughts on the script/evidence as
Now that we've come out of the rabbit hole, we'll leave you with a few parting thoughts:
1. If you ever find yourself really digging into garbage collection in python for data analysis, the odds are that you should probably just be using a bigger machine.
-2. If you're using Hamilton, you should upgrade to the latest version -- it has a lot of improvements and new capabilities, and is optimized to consume less memory.
+2. If you're using Apache Hamilton, you should upgrade to the latest version -- it has a lot of improvements and new capabilities, and is optimized to consume less memory.
3. Garbage collection is *really* interesting, and I just scratched the surface. One of my first mentors in software engineering was a pioneer in the discipline of real-time garbage collection, and I've been trying to soak up as much as I could ever since I first heard about the algorithmic challenges involved.