You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damccorm (via GitHub)" <gi...@apache.org> on 2023/08/04 19:25:30 UTC
[GitHub] [beam] damccorm opened a new pull request, #27857: Add ability to run per key inference
damccorm opened a new pull request, #27857:
URL: https://github.com/apache/beam/pull/27857
This builds on top of #27603 to add support for running inference against a cohort of different models using the KeyedModelHandler. Full design is here - https://docs.google.com/document/d/1kj3FyWRbJu1KhViX07Z0Gk0MU0842jhYRhI-DMhhcv4/edit#heading=h.wskna8eurvjv
This piece adds basic support. In future prs we will need to cover:
- Model updates support
- Metrics improvements
- Example + an E2E test (I have run this in an E2E pipeline, but decided to split the example out from this pr for size reasons)
- Docs
Part of #27628
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291713448
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
Review Comment:
I agree it's less precise; but ultimately users care about the key->model mapping, and MH is an intermediary. I was trying to come up with a more readable and speakable name , esp since it's public api; also it's more common in python to cap all letters in the acronym (e.g. HTTP vs Http, but i i'd rather have MhM than MHM in this case i think...). i don't have a better name though. Maybe others have ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291718410
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
Review Comment:
consider adding a TODO for reference in case someone comes to read the docstring before external reference is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27857: Add ability to run per key inference
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27857:
URL: https://github.com/apache/beam/pull/27857#issuecomment-1666284037
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @AnandInguva for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1290581994
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
Review Comment:
assuming this was moved without changes.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
Review Comment:
Other possible names: KeyModelEntry, KeyModelMapping, KeyMH
Also please add a docstring for this class.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
Review Comment:
we should be able to run predictions for all keys that require the same model before releasing it, or at least iterate over keys in the order that would reduce cache thrashing, WDYT?
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
Review Comment:
given how frequently this appears, consider `if self._single_model` instead
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
Review Comment:
i wonder why this wouldn't work for all modelhandlers. but as is, we'd have an (undocumented?) difference in behavior that we don't delegate estimation to corresponding MHs.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
Review Comment:
Do we plan to adjust batch generation logic to reorganize the keys?
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
Review Comment:
I think this will require more explanations or a "for more information, see: " page that explains how to be careful.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
Review Comment:
i like the name (as opposed to key_to_batch_map). consider similar naming pattern.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
Review Comment:
nit: consider using f-strings, for example.
nit: consider changing order: `f'ModelHandler {mh} defines the following resource hints, which will be ignored: {hints}`
also consider: ` ...by overriding KeyedModelHandler.get_resource_hints() method.`
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
Review Comment:
grammar nit: s/`which`/`, which`
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
Review Comment:
it's not clear what `id` is from reading the code; you did explain this in the design, but you could leave a comment and/or choose a more mnemonic name.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
Review Comment:
you could use: batch_by_key = collections.defaultdict(list)
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
what's the rationale for this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291719157
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
note that this is also a default implementation in the superclass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1294727861
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +245,278 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ """
+ Dataclass for mapping 1 or more keys to 1 model handler.
+ Given `KeyMhMapping(['key1', 'key2'], myMh)`, all examples with keys `key1`
+ or `key2` will be run against the model defined by the `myMh` ModelHandler.
+ """
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
Review Comment:
KeyToModelMapping? or KeyModelMapping?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #27857: Add ability to run per key inference
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #27857:
URL: https://github.com/apache/beam/pull/27857#issuecomment-1666421649
assign to next reviewer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291768724
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
Review Comment:
That's what I ended up with in the design doc so I think I'll stick with it for now. If we come up with something better we do have 6 weeks before the next cut
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291573281
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
Review Comment:
That's right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #27857: Add ability to run per key inference
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #27857:
URL: https://github.com/apache/beam/pull/27857#issuecomment-1666108159
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/27857?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#27857](https://app.codecov.io/gh/apache/beam/pull/27857?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (99bc667) into [master](https://app.codecov.io/gh/apache/beam/commit/887560b35ffd800d5e64e0ba177c1851f69f7f93?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (887560b) will **increase** coverage by `0.02%`.
> Report is 7 commits behind head on master.
> The diff coverage is `95.41%`.
```diff
@@ Coverage Diff @@
## master #27857 +/- ##
==========================================
+ Coverage 70.89% 70.91% +0.02%
==========================================
Files 861 861
Lines 105040 105065 +25
==========================================
+ Hits 74463 74505 +42
+ Misses 29015 28998 -17
Partials 1562 1562
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `79.87% <95.41%> (+0.03%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/beam/pull/27857?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/ml/inference/base.py](https://app.codecov.io/gh/apache/beam/pull/27857?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `94.82% <95.41%> (+0.10%)` | :arrow_up: |
... and [7 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/27857/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291718870
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
will this have an impact on chain calling into unkeyed preprocess funs for single or multi-model case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #27857:
URL: https://github.com/apache/beam/pull/27857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #27857:
URL: https://github.com/apache/beam/pull/27857#issuecomment-1671531523
@tvalentyn FYI, here's an integration test built on top of this PR - https://github.com/apache/beam/pull/27880
I'm keeping it separate for now to keep this one easier to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291713448
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
Review Comment:
I agree it's less precise; but ultimately users care about the key->model mapping, and MH is an intermediary. I was trying to come up with a more readable and speakable name , esp since it's public api; also it's more common in python to cap all letters in the acronym (e.g. HTTP vs Http, but i i'd rather have MhM than MHM in this case i think...). Sadly i don't have a good answer. Maybe others have ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #27857: Add ability to run per key inference
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27857:
URL: https://github.com/apache/beam/pull/27857#issuecomment-1666421830
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @tvalentyn for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1290652238
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
Review Comment:
nit: consider using f-strings.
nit: consider changing order: `f'ModelHandler {mh} defines the following resource hints, which will be ignored: {hints}`
also consider: ` ...by overriding KeyedModelHandler.get_resource_hints() method.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291719157
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
note that this is also a default implementation in the superclass, so we can remove it entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291576601
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
Review Comment:
I agree, I'm planning on addressing that in a full docs pr if that's ok with you - I listed the remaining todos, including calling that out explicitly, in https://github.com/apache/beam/issues/27628
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
Review Comment:
Good idea, done!
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
Review Comment:
> nit: consider using f-strings.
I did this originally and the linter doesn't like it (when using logging.XYZ it prefers `%s` interpolation)
Updated the rest
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
Review Comment:
Added a comment
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
Review Comment:
I think some model handlers can provide tight bounds on batch sizes without doing a full pickling of them. I added shelling out to model handlers as an improvement in #27628 since I think we can may be able to do that per key more efficiently. Or we can just use any of the model handlers since they're all taking the same input type anyways
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
We don't allow these to have a non-zero length anyways (see the constructor). Them being non-empty was pointless before anyways, but I wanted to avoid having to special case the different types of unkeyed here.
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
Review Comment:
I added a docstring, I don't think anything that doesn't have a ModelHandler reference is right though since they're mapping keys to model handlers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #27857: Add ability to run per key inference
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #27857:
URL: https://github.com/apache/beam/pull/27857#discussion_r1291768592
##########
sdks/python/apache_beam/ml/inference/base.py:
##########
@@ -243,70 +244,273 @@ def share_model_across_processes(self) -> bool:
return False
+class _ModelManager:
+ """
+ A class for efficiently managing copies of multiple models. Will load a
+ single copy of each model into a multi_process_shared object and then
+ return a lookup key for that object. Optionally takes in a max_models
+ parameter, if that is set it will only hold that many models in memory at
+ once before evicting one (using LRU logic).
+ """
+ def __init__(
+ self, mh_map: Dict[str, ModelHandler], max_models: Optional[int] = None):
+ """
+ Args:
+ mh_map: A map from keys to model handlers which can be used to load a
+ model.
+ max_models: The maximum number of models to load at any given time
+ before evicting 1 from memory (using LRU logic). Leave as None to
+ allow unlimited models.
+ """
+ self._max_models = max_models
+ self._mh_map: Dict[str, ModelHandler] = mh_map
+ self._proxy_map: Dict[str, str] = {}
+ self._tag_map: Dict[
+ str, multi_process_shared.MultiProcessShared] = OrderedDict()
+
+ def load(self, key: str) -> str:
+ """
+ Loads the appropriate model for the given key into memory.
+ Args:
+ key: the key associated with the model we'd like to load.
+ Returns:
+ the tag we can use to access the model using multi_process_shared.py.
+ """
+ # Map the key for a model to a unique tag that will persist until the model
+ # is released. This needs to be unique between releasing/reacquiring th
+ # model because otherwise the ProxyManager will try to reuse the model that
+ # has been released and deleted.
+ if key in self._tag_map:
+ self._tag_map.move_to_end(key)
+ else:
+ self._tag_map[key] = uuid.uuid4().hex
+
+ tag = self._tag_map[key]
+ mh = self._mh_map[key]
+
+ if self._max_models is not None and self._max_models < len(self._tag_map):
+ # If we're about to exceed our LRU size, release the last used model.
+ tag_to_remove = self._tag_map.popitem(last=False)[1]
+ shared_handle, model_to_remove = self._proxy_map[tag_to_remove]
+ shared_handle.release(model_to_remove)
+
+ # Load the new model
+ shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=tag)
+ model_reference = shared_handle.acquire()
+ self._proxy_map[tag] = (shared_handle, model_reference)
+
+ return tag
+
+ def increment_max_models(self, increment: int):
+ """
+ Increments the number of models that this instance of a _ModelManager is
+ able to hold.
+ Args:
+ increment: the amount by which we are incrementing the number of models.
+ """
+ if self._max_models is None:
+ raise ValueError(
+ "Cannot increment max_models if self._max_models is None (unlimited" +
+ " models mode).")
+ self._max_models += increment
+
+
+# Use a dataclass instead of named tuple because NamedTuples and generics don't
+# mix well across the board for all versions:
+# https://github.com/python/typing/issues/653
+class KeyMhMapping(Generic[KeyT, ExampleT, PredictionT, ModelT]):
+ def __init__(
+ self, keys: List[KeyT], mh: ModelHandler[ExampleT, PredictionT, ModelT]):
+ self.keys = keys
+ self.mh = mh
+
+
class KeyedModelHandler(Generic[KeyT, ExampleT, PredictionT, ModelT],
ModelHandler[Tuple[KeyT, ExampleT],
Tuple[KeyT, PredictionT],
- ModelT]):
- def __init__(self, unkeyed: ModelHandler[ExampleT, PredictionT, ModelT]):
+ Union[ModelT, _ModelManager]]):
+ def __init__(
+ self,
+ unkeyed: Union[ModelHandler[ExampleT, PredictionT, ModelT],
+ List[KeyMhMapping[KeyT, ExampleT, PredictionT, ModelT]]]):
"""A ModelHandler that takes keyed examples and returns keyed predictions.
For example, if the original model is used with RunInference to take a
PCollection[E] to a PCollection[P], this ModelHandler would take a
PCollection[Tuple[K, E]] to a PCollection[Tuple[K, P]], making it possible
- to use the key to associate the outputs with the inputs.
+ to use the key to associate the outputs with the inputs. KeyedModelHandler
+ is able to accept either a single unkeyed ModelHandler or many different
+ model handlers corresponding to the keys for which that ModelHandler should
+ be used. For example, the following configuration could be used to map keys
+ 1-3 to ModelHandler1 and keys 4-5 to ModelHandler2:
+
+ k1 = ['k1', 'k2', 'k3']
+ k2 = ['k4', 'k5']
+ KeyedModelHandler([KeyMhMapping(k1, mh1), KeyMhMapping(k2, mh2)])
+
+ Note that a single copy of each of these models may all be held in memory
+ at the same time; be careful not to load too many large models or your
+ pipeline may cause Out of Memory exceptions.
Args:
- unkeyed: An implementation of ModelHandler that does not require keys.
+ unkeyed: Either (a) an implementation of ModelHandler that does not
+ require keys or (b) a list of KeyMhMappings mapping lists of keys to
+ unkeyed ModelHandlers.
"""
- if len(unkeyed.get_preprocess_fns()) or len(unkeyed.get_postprocess_fns()):
- raise Exception(
- 'Cannot make make an unkeyed model handler with pre or '
- 'postprocessing functions defined into a keyed model handler. All '
- 'pre/postprocessing functions must be defined on the outer model'
- 'handler.')
- self._unkeyed = unkeyed
- self._env_vars = unkeyed._env_vars
-
- def load_model(self) -> ModelT:
- return self._unkeyed.load_model()
+ self._many_models = isinstance(unkeyed, list)
+ if not self._many_models:
+ if len(unkeyed.get_preprocess_fns()) or len(
+ unkeyed.get_postprocess_fns()):
+ raise Exception(
+ 'Cannot make make an unkeyed model handler with pre or '
+ 'postprocessing functions defined into a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ self._env_vars = unkeyed._env_vars
+ self._unkeyed = unkeyed
+ return
+
+ self._id_to_mh_map: Dict[str, ModelHandler[ExampleT, PredictionT,
+ ModelT]] = {}
+ self._key_to_id_map: Dict[str, str] = {}
+ for mh_tuple in unkeyed:
+ mh = mh_tuple.mh
+ keys = mh_tuple.keys
+ if len(mh.get_preprocess_fns()) or len(mh.get_postprocess_fns()):
+ raise ValueError(
+ 'Cannot use an unkeyed model handler with pre or '
+ 'postprocessing functions defined in a keyed model handler. All '
+ 'pre/postprocessing functions must be defined on the outer model'
+ 'handler.')
+ hints = mh.get_resource_hints()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following resource hints %s which will '
+ 'be ignored. Resource hints are not respected when more than one '
+ 'model handler is used in a KeyedModelHandler. If you would like '
+ 'to specify resource hints, you can do so by overriding this '
+ 'KeyedModelHandler and defining the get_resource_hints function.',
+ mh,
+ hints)
+ batch_kwargs = mh.batch_elements_kwargs()
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following batching kwargs %s '
+ 'which will be ignored. Batching kwargs are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. If '
+ 'you would like to specify resource hints, you can do so by '
+ 'overriding this KeyedModelHandler and defining the '
+ 'batch_elements_kwargs function.',
+ hints,
+ batch_kwargs)
+ env_vars = mh._env_vars
+ if len(hints) > 0:
+ logging.warning(
+ 'mh %s defines the following _env_vars %s '
+ 'which will be ignored. _env_vars are not respected when '
+ 'more than one model handler is used in a KeyedModelHandler. '
+ 'If you need env vars set at inference time, you can do so with '
+ 'a custom inference function.',
+ mh,
+ env_vars)
+
+ if len(keys) == 0:
+ raise ValueError(
+ f'Empty list maps to model handler {mh}. All model handlers must '
+ 'have one or more associated keys.')
+ self._id_to_mh_map[keys[0]] = mh
+ for key in keys:
+ if key in self._key_to_id_map:
+ raise ValueError(
+ f'key {key} maps to multiple model handlers. All keys must map '
+ 'to exactly one model handler.')
+ self._key_to_id_map[key] = keys[0]
+
+ def load_model(self) -> Union[ModelT, _ModelManager]:
+ if not self._many_models:
+ return self._unkeyed.load_model()
+ return _ModelManager(self._id_to_mh_map)
def run_inference(
self,
batch: Sequence[Tuple[KeyT, ExampleT]],
- model: ModelT,
+ model: Union[ModelT, _ModelManager],
inference_args: Optional[Dict[str, Any]] = None
) -> Iterable[Tuple[KeyT, PredictionT]]:
- keys, unkeyed_batch = zip(*batch)
- return zip(
- keys, self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return zip(
+ keys,
+ self._unkeyed.run_inference(unkeyed_batch, model, inference_args))
+
+ batch_by_key = {}
+ for key, example in batch:
+ if key not in batch_by_key:
+ batch_by_key[key] = []
+ batch_by_key[key].append(example)
+
+ predictions = []
+ for key, unkeyed_batches in batch_by_key.items():
+ id = self._key_to_id_map[key]
+ mh = self._id_to_mh_map[id]
+ keyed_model_tag = model.load(id)
+ keyed_model_shared_handle = multi_process_shared.MultiProcessShared(
+ mh.load_model, tag=keyed_model_tag)
+ keyed_model = keyed_model_shared_handle.acquire()
+ for inf in mh.run_inference(unkeyed_batches, keyed_model, inference_args):
+ predictions.append((key, inf))
+ keyed_model_shared_handle.release(keyed_model)
+
+ return predictions
def get_num_bytes(self, batch: Sequence[Tuple[KeyT, ExampleT]]) -> int:
- keys, unkeyed_batch = zip(*batch)
- return len(pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ if not self._many_models:
+ keys, unkeyed_batch = zip(*batch)
+ return len(
+ pickle.dumps(keys)) + self._unkeyed.get_num_bytes(unkeyed_batch)
+ return len(pickle.dumps(batch))
def get_metrics_namespace(self) -> str:
- return self._unkeyed.get_metrics_namespace()
+ if not self._many_models:
+ return self._unkeyed.get_metrics_namespace()
+ return 'BeamML_KeyedModels'
def get_resource_hints(self):
- return self._unkeyed.get_resource_hints()
+ if not self._many_models:
+ return self._unkeyed.get_resource_hints()
+ return {}
def batch_elements_kwargs(self):
- return self._unkeyed.batch_elements_kwargs()
+ if not self._many_models:
+ return self._unkeyed.batch_elements_kwargs()
+ return {}
def validate_inference_args(self, inference_args: Optional[Dict[str, Any]]):
- return self._unkeyed.validate_inference_args(inference_args)
+ if not self._many_models:
+ return self._unkeyed.validate_inference_args(inference_args)
+ for mh in self._id_to_mh_map.values():
+ mh.validate_inference_args(inference_args)
def update_model_path(self, model_path: Optional[str] = None):
- return self._unkeyed.update_model_path(model_path=model_path)
+ if not self._many_models:
+ return self._unkeyed.update_model_path(model_path=model_path)
+ if model_path is not None:
+ raise RuntimeError(
+ 'Model updates are currently not supported for ' +
+ 'KeyedModelHandlers with multiple different per-key ' +
+ 'ModelHandlers.')
def get_preprocess_fns(self) -> Iterable[Callable[[Any], Any]]:
- return self._unkeyed.get_preprocess_fns()
+ return []
Review Comment:
> will this have an impact on chain calling into unkeyed preprocess funs for single or multi-model case?
Yeah, though not significantly. It doesn't allow you to strictly chain unkeyed functions if you have a KeyedModelHandler, though you can pretty easily do things like: `with_preprocess_fn(lambda k, v: k, my_func(v))`. Note that this PR doesn't change that behavior
> note that this is also a default implementation in the superclass, so we can remove it entirely
Good call, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org