[MLlib] Differences after version upgrade

[Roger Wechsler <mr...@gmail.com>]

Hi!

We've been using Spark 3.0.1 to train Logistic regression models with MLLIb.
We've recently upgraded to Spark 3.3.0 without making any other code
changes and noticed that the trained models are different as compared to
the ones trained with 3.0.1 and therefore behave differently when used for
prediction.

We went through the release notes and through the API changes to see if the
default behaviour changed, but we could not find anything. Do you know what
changed between version 3.01 and 3.3.0? And if so, how could we guarantee
the same behaviour as in 3.0.1?

Thanks for your help!

Roger

Re: [MLlib] Differences after version upgrade

[Sean Owen <sr...@gmail.com>]

How different? I think quite small variations are to be expected.

On Wed, Jul 20, 2022 at 9:13 AM Roger Wechsler <mr...@gmail.com> wrote:

> Hi!
>
> We've been using Spark 3.0.1 to train Logistic regression models
> with MLLIb.
> We've recently upgraded to Spark 3.3.0 without making any other code
> changes and noticed that the trained models are different as compared to
> the ones trained with 3.0.1 and therefore behave differently when used for
> prediction.
>
> We went through the release notes and through the API changes to see if
> the default behaviour changed, but we could not find anything. Do you know
> what changed between version 3.01 and 3.3.0? And if so, how could we
> guarantee the same behaviour as in 3.0.1?
>
> Thanks for your help!
>
> Roger
>