You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@hamilton.apache.org by "eric-czech (via GitHub)" <gi...@apache.org> on 2025/07/30 16:57:58 UTC
[I] Add a dry run feature [hamilton]
eric-czech opened a new issue, #1370:
URL: https://github.com/apache/hamilton/issues/1370
**Is your feature request related to a problem? Please describe.**
It would be helpful to know what steps of a DAG *would* execute when caching is enabled.
**Describe the solution you'd like**
For an example like [this](https://hamilton.staged.apache.org/concepts/caching/#visualization), being able to visualize DAGs or query cache results prior to execution through any Driver function would be great.
I imagine I'm far from the first person to think about this, but I'm not seeing any prior discussion about it. Are there any foundational design issues that would make this difficult to implement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a dry run feature [hamilton]
Posted by "elijahbenizzy (via GitHub)" <gi...@apache.org>.
elijahbenizzy commented on issue #1370:
URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3162397702
@eric-czech -- nothing that would make this impossible! It's quite doable. In fact, we have https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.visualize_execution which should get you most of the way there!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a dry run feature [hamilton]
Posted by "eric-czech (via GitHub)" <gi...@apache.org>.
eric-czech commented on issue #1370:
URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3331115324
Thanks @zilto, great explanation! That diagram does help a lot.
All use cases I have in mind would involve passing nothing other than URIs to paths on cloud storage systems between Hamilton steps, and they would all be known prior to DAG execution. I.e. my outputs would all be individual strings that rarely change or change in predictable ways (e.g. based on dates or versions). That would be far more amenable to a dry-run, but I understand what you're saying that hamilton is supporting content-based, runtime dependencies.
It would be great if something like `visualize_execution` could run prior to execution for DAGs that make stronger assumptions somehow. I am essentially asking for `--dry-run` as it works in `make` since that's more or less how I would like to use hamilton. Or snakemake might be a better comparison given that it's a bit more cloud native.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a dry run feature [hamilton]
Posted by "zilto (via GitHub)" <gi...@apache.org>.
zilto commented on issue #1370:
URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3246451294
Actually, it is *likely to be impossible* to view "what nodes will be executed before launching execution when using caching".
In short, the caching feature needs to "walk the graph". You start at input nodes and follow the execution path until the final nodes. At each step, caching checks the `code_version` of the current node and the `data_version` of its dependencies.
Before executing the DAG, you don't know the `data_version` of all nodes (if you did, you wouldn't need to execute the DAG at all!). Before executing the DAG, you're able to make 3 types of claims:
- "Will be retrieved from cache"
- "Might be retrieved from cache"
- "Will need to be executed / will not be retrieved from cache"
Visually:
<img width="1435" height="990" alt="Image" src="https://github.com/user-attachments/assets/d0cdfa44-bb6c-49f2-845e-a79f803bc312" />
The thing is, whenever you hit a "maybe will be retrieved from cache", everything downstream of it will necessarily be a "maybe" too. Since what changes between runs are the input nodes (i.e., top-level nodes), you will find that you almost always have visualizations filled with maybes. Though, the viz could be useful is certain specific topologies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a dry run feature [hamilton]
Posted by "zilto (via GitHub)" <gi...@apache.org>.
zilto commented on issue #1370:
URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3285205955
## How caching works
Before execution, you are certain of your input `data_version` and the `code_version` of **all nodes**. What you don't know is the `data_version` of **all non-input nodes**. To know the `data_version`, you need to apply the caching logic by traversing the DAG.
In DAG `A -> B -> C -> D`, to know if `D` needs to be executed, you need to resolve `C.data_version` (you already know `C.code_version`). To know `C.data_version`, you need to resolve `B.data_version`. The key point being, you need to resolve things in order.
(The best summary I can provide is the picture in the previous reply)
## In short
> Put differently, couldn't you provide a dry-run feature that works under the assumption that all data versions will be the same if all upstream code versions have not changed?
As I understand, the goal of the visualization is to tell you "how caching will behave before running the DAG". As explained in a previous reply, it **is** possible, but almost all your nodes will be "maybe execute".
If this visualization makes different assumptions than the caching algorithm, it will display something that diverges from the actual caching behavior. The output will be probably be more confusing than useful.
## Assumptions
### dry-run
> under the assumption that all data versions will be the same if all upstream code versions have not changed?
If you already know the `code_version` and `data_version` of A, B, C, D, there's no need to execute the DAG. A `data_version` is a hash of a specific Python object. If you know `D.data_version = XYZ123`, then just `cache.get(D.data_version)` and retrieve the value.
If you do know that `B = 4`, then you should use `Driver.execute(["D"], overrides={"B": 4})`. Caching will properly handle this knowledge. If you were to implement the visualization I explained in a previous reply, you would see less "maybe execute", but still potentially a lot of them.
## Idempotency
> What is assumed to be capable of changing data if not code though?
The same code can produce different results (non-idempotency): it has a stochastic component (`random.choice`, ML training), it does I/O and reads values from an external file, you use a different version of an external Python library (hypothetically, `numpy.float` defaulting to 32 precision in `numpy==2.0.0` vs. 64 precision in `numpy==3.0.0`).
By default Hamilton assumes idempotency of nodes and it's the user's responsibility to tag non-idempotent nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a dry run feature [hamilton]
Posted by "eric-czech (via GitHub)" <gi...@apache.org>.
eric-czech commented on issue #1370:
URL: https://github.com/apache/hamilton/issues/1370#issuecomment-3282946492
> you don't know the data_version of all nodes
That makes sense, thanks @zilto. What is assumed to be capable of changing data if not code though? Put differently, couldn't you provide a dry-run feature that works under the assumption that all data versions will be the same if all upstream code versions have not changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org