You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@hamilton.apache.org by "gidhubuser255 (via GitHub)" <gi...@apache.org> on 2025/10/01 20:50:30 UTC

[D] APIs for updating underlying function of a node (2 part question) [hamilton]

GitHub user gidhubuser255 created a discussion: APIs for updating underlying function of a node (2 part question)

### (1) Updating function logic
For reference consider the following DAG

```
def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D() -> int:
    return 7

dr = driver.Builder().with_modules(__import__('__main__')).build()
dr.execute('A')  # returns 17
```

If I want to update the logic of `B` in the DAG with some function `alternative_B`, the best way I've found to do so is to do the following:
```
@config.when(use_alternative_b=False)
def B(D: int) -> int:
    return D + 1

@config.when(name='B', use_alternative_b=True)
def alternative_B(C: int, D: int) -> int:
    return C + D

dr = (
    driver.Builder()
    .with_modules(__import__('__main__'))
    .with_config(use_alternative_b=True)
    .build()
)
dr.execute('A')  # returns 25
```
Which is a bit cumbersome, since it requires modifying source code and adding configuration primitives to enable the ability.

An alternative way to do this is how the fn_graph library handles this (https://fn-graph.readthedocs.io/en/latest/usage.html#building-up-logic), which provides a similar ability to update the implementation of a node by allowing you to make updates on your "driver" (called composer in fn_graph). For example to achieve the same ability in fn_graph you can do this, no decorators or additional config entities needed:
```
composer = Composer().update(A, B, C, D)  # these are all the same defined functions as above
composer2 = composer.update(B=alternative_B)  # just plain defined function, no decorators
composer3 = composer2.update(B=alternative_B2)

composer2.A()
composer3.A()
```
**Questions:**
1. Is there a similar existing mechanism to this in Hamilton?
2. If not could it be added?

### (2) Updating function logic in a specific branch
Similar to the above consider when you want to update the implementation for a function, but only when the path from that function to the root executed function passes through another specific node/function.

For example if I want to update `D` to `D__prime` but only in the path that includes `B`, in Hamilton I could achieve that like so:
```
def A(B: int, C: int) -> int:
    return B + C

def B(D_thru_B: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

@config.when(use_d_prime=False)
def D_thru_B(D: int) -> int:
    return D

@config.when(name='D_thru_B', use_d_prime=True)
def D__prime(D: int) -> int:
    return D * 2

def D() -> int:
    return 7
```
But this has a couple problems: (a) if I want to later update all references to `D` with `D__prime` (not just through `B`) I'll need to add another `D__prime2` with a `config.when(name='D', ...)` and a new config parameter, which is again pretty cumbersome. And more importantly (b) this method breaks down when the branch specific replacement happens behind a converging point, for example:
```
def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D(X: int) -> int:
    return X + 1

def X() -> int:
    return 9
```
If I want to replace `X` in the path of `B` with this method I would need to bifurcate all my functions starting at the converging point (`D_thru_B`, `X_thru_B`, `D_thru_C`, `X_thru_C`, etc), which would be prohibitive for anything more than a very simple example.

**Questions:**
1. Is there a good existing way to achieve this? I'm thinking there's probably a way using the existing `Parallelizable` and `Collect` mechanism which can already dynamically generate new nodes to add to the graph, and which could be used to do the bifurcation I mentioned?
2. Would it be possible to add the ability to do this with something more like the fn_graph style api, something like `driver.update(X=X__prime, in_path=['B']) `


GitHub link: https://github.com/apache/hamilton/discussions/1397

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> What does "override" refer to here? 

Overrides is what we can do to an instantiated driver object. Right now we can provide values that override a node.

What you suggest here would be a form of it:

> driver = driver.update(date=new_date, in_path=['shipment_view'])
> driver.execute('main_view')


> One other way I could see this working which is more consistent with @config.when for node replacement, would be something like @path.passes_through, which could work something like this:
> 
> def X() -> int:
>     return 9
> 
> @path.passes_through('B', name='X')
> def X_prime() -> int:
>     return 99

I see. So this feels like you just want to avoid having to update B's parameters for this to work. Since I feel it's not that many more characters to update B's parameter and not have the decorator here.

I think I'd want this instead:

```python

@path.passes_through(B, name=X) # could we do references? or would that be too constraining?
def X_prime() -> int:
     return 99
```

Edge cases to think about:
1. how would one debug this? What would show up in the "viz" view?
2. Is the main use case to apply updates given a graph?  Since I think if it's static code, I'd just argue that people should make the edits...
3. could a LSP server help stitch together autocomplete / a view of the graph easily? (we haven't put much time into the vscode plugin, but we could...)

I like this:

> driver = driver.update(date=new_date, in_path=['shipment_view'])

and/or combining it with the above
```
# some module
@path.passes_through('B', name='X')
def X_prime() -> int:
     return 99
```

After all Hamilton is just nodes and edges. So "graph chaining" and this type of "graph overrides/transformation" doesn't seem out of the realm of what Hamilton could support.



GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14589291

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

@skrawcz assuming my above assumption is correct, is there some process for adding feature requests to a backlog or roadmap, or opening a wider discussion for adoption? Thanks.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14629668

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz edited a comment on the discussion: APIs for updating underlying function of a node (2 part question)

> Personally I do prefer the driver.update method as it makes the ephemeral cases easier to work with, but I realize that is a bit more complicated to implement given the module based architecture. Plus I think I have some dynamic module magic (using types.ModuleType) I can mess with to get it to work closer to driver.update.

Actually once in the driver, we have the nodes & edges data structure, so modules don't play into this anymore -- we already have pointers to the functions.

E.g. think of a driver as an instantiated graph. We can then perform graph transformations on it that result in another static graph. 

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14592095

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 edited a comment on the discussion: APIs for updating underlying function of a node (2 part question)

### (1)
The override modules method worked thanks! Still would be nice to manage with function overrides directly on the driver, but I see there is some relevant work being discussed here (https://github.com/apache/hamilton/issues/1268) already that might get that closer to reality. (Also looks like I can get some pseudo-functionality of this using types.ModuleType to create custom module objects that I can make updates on.)

Would it be acceptable to re-request this (if not already implemented) when the work on first class function instantiation is more mature?

### (2)
To expand on part 2 take a look at these 2 graphs
<img width="579" height="290" alt="image" src="https://github.com/user-attachments/assets/fd00c884-debb-4725-a711-3c4b87ceb6ea" />

The left graph is the original DAG structure as defined by the functions. What I want to do is update X -> X' but only when considering the path that passes through B. Doing so will result in the right graph. You'll notice in the right graph the D in the 2 branches are identical even though they are on different branches. So essentially the idea is that I'm looking for some mechanism that can unconverge paths into separate paths, without having to manually define each separate path since this can explode exponentially depending on how many converging points you have and which in_paths you want to update in. 

Subdag and pipe look promising I'll take a deeper look at those, thanks! However, one downside off the bat I can think of is that the model will need to be pre-constructed/structure hardcoded for pre-determined unraveled branches, as opposed to having a mechanism that can dynamically unravel branches for these any path-specific updates, like the one proposed.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14569798

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

@skrawcz - Hi, following up on this:

>  is there some process for adding feature requests to a backlog or roadmap

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14840147

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz edited a comment on the discussion: APIs for updating underlying function of a node (2 part question)

thanks. Not sure I fully grok (2) yet,  but just to mention Hamilton's main design principle is static-ness. Why? because it's easier to debug; sure it's more verbose, but it's explicit and you're more likely to understand someone else's code.

Now I think there could be something here to help support a graph transform to functionally get what you want..

But I'd argue:
1. the Ds are not the same on the right. They are a function of their inputs and thus their result is different.
2. So from a debugging and understanding standpoint, if they are to be executed together then they are separate nodes IMO.
3. Which in Hamilton requires you to create the graph on the right -- you can reuse logic, but structurally you need to declare the right.

That said, applying some transform from left to right that doesn't seem un-reasonable. So this is where your input and ideas would be needed. I see you did already mention some APIs (thanks!) -- some questions that come to mind:
1. is this for quick ad-hoc stuff?
2. or structurally you want to swap things like this out in "production"?
3. or?

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14570575

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz edited a comment on the discussion: APIs for updating underlying function of a node (2 part question)

> What does "override" refer to here? 

Overrides is what we can do to an instantiated driver object. Right now we can provide values that override a node.

What you suggest here would be a form of it:

> driver = driver.update(date=new_date, in_path=['shipment_view'])
> driver.execute('main_view')


> One other way I could see this working which is more consistent with @config.when for node replacement, would be something like @path.passes_through, which could work something like this:
> 
> def X() -> int:
>     return 9
> 
> @path.passes_through('B', name='X')
> def X_prime() -> int:
>     return 99

I see. So this feels like you just want to avoid having to update B's parameters for this to work. Since I feel it's not that many more characters to update B's parameter and not have the decorator here.

I think I'd want this instead:

```python

@path.passes_through(B, name=X) # could we do references? or would that be too constraining?
def X_prime() -> int:
     return 99
```

Edge cases to think about:
1. how would one debug this? What would show up in the "viz" view?
2. Is the main use case to apply updates given a graph?  Since I think if it's static code that is checked in and not touched again, I'd just argue that people should make the edits...
3. could a LSP server help stitch together autocomplete / a view of the graph easily? (we haven't put much time into the vscode plugin, but we could...)

I like this:

> driver = driver.update(date=new_date, in_path=['shipment_view'])

and/or combining it with the above
```
# some module
@path.passes_through('B', name='X')
def X_prime() -> int:
     return 99
```

After all Hamilton is just nodes and edges. So "graph chaining" and this type of "graph overrides/transformation" doesn't seem out of the realm of what Hamilton could support.



GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14589291

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> we could modify override that it does 2 more things

What does "override" refer to here? Is that a decorator or method?

One other way I could see this working which is more consistent with `@config.when` for node replacement, would be something like `@path.passes_through`, which could work something like this:

```
def X() -> int:
    return 9

@path.passes_through('B', name='X')
def X_prime() -> int:
    return 99
```

With regards to your pseudo-code example of how to make that work currently, that does indeed work, however gets a bit unwieldy for anything beyond a simple example. If you have multiple nodes between your converging point and the node you want to replace, you'll have to enumerate them all as steps in your pipe_inputs, like so:

```
from hamilton import driver
from hamilton.function_modifiers import pipe_input, step

def A(B: int, C: int) -> int:
    return B + C

@pipe_input(
    step(Y),
    step(Z),
    step(D),
)
def B(X: int) -> int:
    return X + 1

@pipe_input(
    step(Y),
    step(Z),
    step(D),
)
def C(X: int) -> int:
    return X + 2

def D(Z: int) -> int:
    return Z + 1

def Z(Y: int) -> int:
    return Y * 2

def Y(X: int) -> int:
    return X * 3

def X() -> int:
    return 9

dr = (
    driver.Builder()
    .with_modules(__import__('__main__'))
    .build()
)

print(dr.execute('A'))
```

And if there's more bifurcating/converging points in between that gets significantly harder to represent.

> how this will play with caching

Unfortunately I'm not familar with how the caching scheme works in depth. However, my hunch would be that since this would still allow a static built graph (just with an extra post processing step that would analyze the paths and unravel them if necessary) it could potentially be compatible with the existing caching scheme.


GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14587007

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "jernejfrank (via GitHub)" <gi...@apache.org>.
GitHub user jernejfrank added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

Hey, cool ideas.

> ### (1) Updating function logic
> For reference consider the following DAG
> 
> ```
> def A(B: int, C: int) -> int:
>     return B + C
> 
> def B(D: int) -> int:
>     return D + 1
> 
> def C(D: int) -> int:
>     return D + 2
> 
> def D() -> int:
>     return 7
> 
> dr = driver.Builder().with_modules(__import__('__main__')).build()
> dr.execute('A')  # returns 17
> ```

> **Questions:**
> 
> 1. Is there a similar existing mechanism to this in Hamilton?
> 2. If not could it be added?
> 

At the moment Hamilton does everything via modules (there's things brewing to enable directly passing functions in, but not there yet).  There is a way to override `B` by defining it in a seperate modue and then using `.allow_module_overrides()` during the building process (checkout this [example](https://github.com/apache/hamilton/tree/main/examples/module_overrides). In your case:

```python
# new_module.py

@config.when(name='B', use_alternative_b=True)
def B(C: int, D: int) -> int:
    return C + D
```

and

```python
# main.py

def A(B: int, C: int) -> int:
    return B + C

def B(D: int) -> int:
    return D + 1

def C(D: int) -> int:
    return D + 2

def D() -> int:
    return 7

if __name__ == "__main__":
    import __main__ as main
    import new_module

    dr = driver.Builder().with_modules(main, new_module).allow_module_overrides().build()
    dr.execute('A')
```

A word of warning, if you use multiple modules and have in them defined functions with the same name Hamilton will throw an error. However, with the `.allow_module_overrides()` flag order matters `.with_modules(main, new_module) != .with_modules(new_module,main)` and the flag is global so in case you use multiple modules you need to be careful not to override something you don't want to.


> ### (2) Updating function logic in a specific branch

> **Questions:**
> 
> 1. Is there a good existing way to achieve this? I'm thinking there's probably a way using the existing `Parallelizable` and `Collect` mechanism which can already dynamically generate new nodes to add to the graph, and which could be used to do the bifurcation I mentioned?
> 2. Would it be possible to add the ability to do this with something more like the fn_graph style api, something like `driver.update(X=X__prime, in_path=['B']) `

I am not sure a fully understand the second part,  but I can give it a go and let me know if I missed something. Hamilton will create the full graph, but it will only walk / execute nodes that are relevant for a specified output (in your case `A`). You cannot really control which branch of the graph will execute, only the nodes that are leading to the requested output. In your case, `B` will always execute because `A` needs it (same for `C`) all the way down to `X`. What you have control over is what the individual nodes are and there's a couple of ways to organise your code to manage multiple potential branches:

1. Define "static" nodes (things that you think will not change) in a seperate module and organise "dynamic" functions into different modules. You can then during the build time choose which module to load that will create the desired nodes. like having a feature catalog.
2. Using `subdag` decorator to re-use functions [example](https://github.com/apache/hamilton/tree/main/examples/reusing_functions) and run custom tags esentially as a node.
3. Similar to subdag for simpler stuff maybe the `pipe` family might be easier to use [example](https://github.com/apache/hamilton/tree/main/examples/scikit-learn/species_distribution_modeling)

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14567925

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> @path.passes_through(B, name=X) # could we do references?

I think using a reference to the function could cause problems because if B is replaced with B_prime under the hood this path decorator would not trigger, when it actually should.

Regarding your edge cases:

(1) I would imagine this would look similar to the viz of config.when in driver.visualize_execution, where there is some suffix on the node name. So something like this:
<img width="530" height="277" alt="image" src="https://github.com/user-attachments/assets/87218cdf-82ed-4ad5-8cbe-28635d4fb939" />

(2) The way I see it being used is mostly ephemeral replacements for things like my UI stuff I mentioned as well as sensitivity analysis in computational models. Although I have seen static logic replacements like this (not using Hamilton) used in large models where the complexity would have been prohibitive to manually code the unraveled paths (but many of those cases could probably have been considered bad design anyway).

(3) Can't speak to this atm, as I'm unfamiliar with the capabilities and requirements of LSP servers

Personally I do prefer the driver.update method as it makes the ephemeral cases easier to work with, but I realize that is a bit more complicated to implement given the module based architecture. Plus I think I have some dynamic module magic (using types.ModuleType) I can mess with to get it to work closer to driver.update.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14590108

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> so modules don't play into this anymore ... We can then perform graph transformations on it that result in another static graph.

Oh nice, so going back to just part (1) of my question (for non-path-dependent updates), seems like no major architectural changes would be needed to expose some kind of mechanism like `driver.update(x=x_prime)` (with no in_path argument)? 

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14592571

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 edited a comment on the discussion: APIs for updating underlying function of a node (2 part question)

### (1)
The override modules method worked thanks! Still would be nice to manage with function overrides directly on the driver, but I see there is some relevant work being discussed here (https://github.com/apache/hamilton/issues/1268) already that might get that closer to reality. (Also looks like I can get some pseudo-functionality of this using types.ModuleType to create custom module objects that I can make updates on.)

I'll follow back up on this if not already implemented when the work on first class function instantiation is more mature

### (2)
To expand on part 2 take a look at these 2 graphs
<img width="579" height="290" alt="image" src="https://github.com/user-attachments/assets/fd00c884-debb-4725-a711-3c4b87ceb6ea" />

The left graph is the original DAG structure as defined by the functions. What I want to do is update X -> X' but only when considering the path that passes through B. Doing so will result in the right graph. You'll notice in the right graph the D in the 2 branches are identical even though they are on different branches. So essentially the idea is that I'm looking for some mechanism that can unconverge paths into separate paths, without having to manually define each separate path since this can explode exponentially depending on how many converging points you have and which in_paths you want to update in. 

Subdag and pipe look promising I'll take a deeper look at those, thanks! However, one downside off the bat I can think of is that the model will need to be pre-constructed/structure hardcoded for pre-determined unraveled branches, as opposed to having a mechanism that can dynamically unravel branches for these any path-specific updates, like the one proposed.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14569798

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

thanks. Not sure I fully grok (2) yet,  but just to mention Hamilton's main design principle is static-ness. Why? because it's easier to debug; sure it's more verbose, but it's explicit and you're more likely to understand someone else's code.

Now I think there could be something here to help support a graph transform to functionally get what you want..

But I'd argue:
1. the Ds are not the same on the right. They are a function of their inputs and thus their result is different.
2. So from a debugging and understanding standpoint, if they are to be executed together then they are separate nodes IMO.
3. Which in Hamilton requires you to create the graph on the right -- you can reuse logic, but structurally you need to declare the right.

That said, applying some transform from left to right that doesn't seem un-reasonable. So this is where your input and ideas would be needed. I see you did already mention some APIs (thanks!) -- some questions that come to mind:
1. is this for quick ad-hoc stuff?
2. or structurally you want to swap things like this out?
3. or?

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14570575

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> the Ds are not the same on the right.

Agreed, within the wider context of the full graph (and more specifically the upstream node differences). To clarify, what I was referring to was, for the node in isolation the underlying functions/computation logic is identical. The reason I pointed this out was to draw attention to the fact that even though they are identical in isolation, they can not be currently represented in hamilton by a single static function because they have different upstream dependencies. So they either need to be represented as separate static functions (e.g. my enumerated X_thru_Y examples in the OP), or some mechanism would need to take a single function and apply its logic to separate nodes in the graph (which I will look into subdag etc to see how those can help).

To expand on my immediate use case I'm currently working on building a web-UI framework with hamilton as the underlying engine. Imagine this contrived pseudo-code/pseudo-typed DAG:

```
def main_view(orders_view: HTMLString, shipments_view: HTMLString) -> HTMLString:
    return f'''
    {orders_view}
    {shipments_view}
    '''

def orders_view(orders: List[Order]) -> HTMLString:
    # includes orders date picker
    ...

def shipments_view(shipments: List[Shipment]) -> HTMLString:
    # include shipments date picker
    ...

def orders(date: Date) -> List[Order]:
    ...

def shipments(orders: List[Order], date: Date) -> List[Shipment]:
    ...

def date() -> Date:
    ...
```

Now let's say in my UI I want to select a new date for the shipments view. I don't want my orders view to be affected, but they share upstream components so currently there's no way to just take the above defined functions and get the behavior I want. With my proposed api I could just do this: 

```
driver = driver.update(date=new_date, in_path=['shipment_view'])
driver.execute('main_view')
```

Besides this UI example the path specific update concept is very useful in identifying sensitivities and error sources in timeseries forecasting.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14581669

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "jernejfrank (via GitHub)" <gi...@apache.org>.
GitHub user jernejfrank added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

Ok I think I got it now. Atm i don't think we have any mechanism to modify the graph after it has been created, just a lot of helper decorators for creating the DAGs (that let you re-use functions). The closest you can come is with config/overrides during the execution to switch things out dynamically, but IIRC overrides can only inject values instead of nodes not actually replace the callable in the node.

A pseudo code example to achieve this at the moment would be

```python
def _D(): ...

def X(...): ...

@pipe_input(
    step(_D, ...),
)
def B(X,...): ...

@pipe_input(
    step(_D, ...),
)
def C(X,...): ...
```

Now, if you need to update `X-->X'` in path through B, you have

```python
def _D(): ...

def X(...): ...

def X'(...): ...

@pipe_input(
    step(_D, ...),
)
def B(X',...): ...

@pipe_input(
    step(_D, ...),
)
def C(X,...): ...
```

Your request is very interesting tho. To my understanding, we could modify override that it does 2 more things:
- accepts a callable that replaces the current callable in the node (i.e. override a whole node with need functionality)
- add a second flag `path_through` that looks at the existing graph and creates a new path from the override node to the target node specified by `path_through` and duplicates those nodes and wires them correctly to not conflict with the other branches.

The only thing I am not sure about is how this will play with caching. Am I missing something else?

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14582769

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "gidhubuser255 (via GitHub)" <gi...@apache.org>.
GitHub user gidhubuser255 added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

### (1)
The override modules method worked thanks! Still would be nice to manage with function overrides directly on the driver, but I see there is some relevant work being discussed here (https://github.com/apache/hamilton/issues/1268) already that might get that closer to reality. (Also looks like I can get some pseudo-functionality of this using types.ModuleType to create custom module objects that I can make updates on.)

Would it be acceptable to re-request this (if not already implemented) when the work on first class function instantiation is more mature?

### (2)
To expand on part 2 take a look at these 2 graphs
<img width="579" height="290" alt="image" src="https://github.com/user-attachments/assets/fd00c884-debb-4725-a711-3c4b87ceb6ea" />

The left graph is the original DAG structure as defined by the functions. What I want to do is update X -> X' but only when considering the path that passes through B. Doing so will result in the right graph. You'll notice in the right graph the D in the 2 branches are identical even though they are on different branches. So essentially the idea is that I'm looking for some mechanism that can unconverge paths into separate paths, without having to manually define each separate path since this can explode exponentially depending on how many converging points you have and which in_paths you want to update in. 

Subdag and pipe look promising I'll take a deeper look at those. However, one downside off the bat I can think of is that the model will need to be pre-constructed/structure hardcoded for pre-determined unraveled branches, as opposed to having a mechanism that can dynamically unravel branches for these any path-specific updates, like the one proposed.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14569798

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org


Re: [D] APIs for updating underlying function of a node (2 part question) [hamilton]

Posted by "skrawcz (via GitHub)" <gi...@apache.org>.
GitHub user skrawcz added a comment to the discussion: APIs for updating underlying function of a node (2 part question)

> Personally I do prefer the driver.update method as it makes the ephemeral cases easier to work with, but I realize that is a bit more complicated to implement given the module based architecture. Plus I think I have some dynamic module magic (using types.ModuleType) I can mess with to get it to work closer to driver.update.

Actually once in the driver, we have the nodes & edges data structure, so modules don't play into this anymore -- we already have pointers to the functions.

GitHub link: https://github.com/apache/hamilton/discussions/1397#discussioncomment-14592095

----
This is an automatically sent email for notifications@hamilton.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@hamilton.apache.org