[GitHub] [beam] TheNeuralBit commented on issue #22572: [Feature Request]: Allow specification of a custom model inference method for a RunInference ModelHandler

[GitBox <gi...@apache.org>]

TheNeuralBit commented on issue #22572:
URL: https://github.com/apache/beam/issues/22572#issuecomment-1204540478

   I think this would be difficult to do in a general (cross-ModelHandler) way as each ModelHandler is responsible for invoking it's model, and they currently have different ways of doing so.
   
   sklearn calls a predict method: https://github.com/apache/beam/blob/5b1e1520b975de563b8b57144927894a2fddded1/sdks/python/apache_beam/ml/inference/sklearn_inference.py#L124
   
   pytorch calls the model like a callable (which then uses the forward method IIUC?): https://github.com/apache/beam/blob/5b1e1520b975de563b8b57144927894a2fddded1/sdks/python/apache_beam/ml/inference/pytorch_inference.py#L235
   
   I think the best we could do to solve the problem generally is establish some kind of convention.
   
   It's also worth noting that the `generate` method is a property of hugging face's `GenerationMixin`, not a part of the `torch.nn.Module` API, which is in our contract: https://github.com/apache/beam/blob/5b1e1520b975de563b8b57144927894a2fddded1/sdks/python/apache_beam/ml/inference/pytorch_inference.py#L199
   
   Is a separate generation modelhandler a better solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org