Xgboost inference on regression task.

[askelejboelle <as...@veovo.com>]

Hi,

I have trained an xgboost model in python to perform prediction of
continuous numerical values.
However, when I load the model into ignite and perform predictions, they are
bounded to [0,1]. Does ignite convert predicted values and is prediction on
regression tasks currently supported? I formatted my model and data similar
to the ignite example.
 
Here's my model, for the sake of simplicity, I trained a model containing
two trees and a max depth of 3:


Here's my input feature vector:
1:57 2:321 3:285 4:23 5:0 6:44 7:2 8:1 9:1 10:1 11:1 12:675 13:4 14:19

Expected prediction: 1.131945495605468750e+02 (prediction from Python)
Prediction: 1.0

Thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Xgboost inference on regression task.

[zaleslaw <za...@gmail.com>]

Hi, askelejboelle!

Could you please share the model, snippet of data (if it's not secret) and
full code sample to reproduce problem. It looks like a bug and I'll try to
fix it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Xgboost inference on regression task.

[zaleslaw <za...@gmail.com>]

Dear akorensh, thank you for the support, please, next time provide the
correct link to Apache Ignite documentation, not to GridGain site (there
could be outdated documentation).

The actual documentation is here for release 2.8.0
https://apacheignite.readme.io/docs/machine-learning

https://apacheignite.readme.io/docs/gradient-boosting
https://apacheignite.readme.io/docs/random-forest





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Xgboost inference on regression task.

[akorensh <al...@gmail.com>]

Hello,
  At Present Ignite does convert the predicted values for XGBoost models as
follows: 
            double res = 0;

            for (double prediction : predictions)
                res += prediction;

            return (1.0 / (1.0 + Math.exp(-res)));

More flexible aggregations for XGBoost are coming in the future.



You can use the model for classification/regression depending on how you
encode it, bearing in mind the aggregation implementation noted above.

Take a look at Ignite's native gradient boosting capabilities:
https://www.gridgain.com/docs/latest/developers-guide/machine-learning/grad-boost

classification:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/tree/boosting/GDBOnTreesClassificationTrainerExample.java

regression:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/ml/tree/boosting/GDBOnTreesRegressionTrainerExample.java

GDBModel uses WeightedPredictionsAggregator as the model answer reducer.
This aggregator computes an answer of a meta-model, since "result = bias +
p1w1 + p2w2 + …​" where:

pi - answer of i-th model.

wi - weight of model in composition.

GDB uses the mean value of labels for the bias-parameter in the aggregator.


Also take a look at Ignite's Random Forest learner:
https://www.gridgain.com/docs/latest/developers-guide/machine-learning/random-forest

It provides a choice if  Aggregators depending on the problem domain:

MeanValuePredictionsAggregator - computes answer of a random forest as mean
value of predictions from all models in the given composition. Often this is
used for regression tasks.

OnMajorityPredictionsAggegator - gets a mode of predictions from all models
in the given composition. This can be useful for a classification task.
NOTE: This aggregator supports multi-classification tasks.


Thanks, Alex




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/