You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/02/16 23:07:43 UTC
[GitHub] aosagie opened a new pull request #23813: [SPARK-26721][ML] Remove
per tree feature importance normalization for gbt classifier/regressor
aosagie opened a new pull request #23813: [SPARK-26721][ML] Remove per tree feature importance normalization for gbt classifier/regressor
URL: https://github.com/apache/spark/pull/23813
## What changes were proposed in this pull request?
It was discovered that scikit learn was miscalculating GBT feature importances due to mistakenly normalizing each individual tree. This fixes the issue in SparkML (which appears to have followed what scikit learn did). See: https://github.com/scikit-learn/scikit-learn/pull/11176
## How was this patch tested?
Manually tested by running the following script:
```python
import pandas
from sklearn.datasets import fetch_california_housing
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import GBTRegressor, RandomForestRegressor
california = fetch_california_housing()
pandas_df = pandas.DataFrame(california.data, columns=california.feature_names)
pandas_df["label"] = pandas.Series(california.target)
df = spark.createDataFrame(pandas_df)
train, test = df.randomSplit([.75, .25], seed=0)
train2 = VectorAssembler(inputCols=california.feature_names, outputCol="features").transform(train)
gbt = GBTRegressor(seed=0, lossType="absolute", maxDepth=3)
gbt_model = gbt.fit(train2)
print(sorted(zip(california.feature_names, gbt_model.featureImportances), key=lambda tup: -tup[1]))
#Before Change: [('Longitude', 0.2581418258949404), ('Latitude', 0.2558924988641387), ('MedInc', 0.24361394155329505), ('AveOccup', 0.11946847304433225), ('HouseAge', 0.07752951696478831), ('AveBedrms', 0.02594190009061629), ('AveRooms', 0.01941184358788898), ('Population', 0.0)]
#After Change: [('MedInc', 0.40777614558392367), ('Longitude', 0.20928977828611595), ('Latitude', 0.18723522315674387), ('AveOccup', 0.12284687396572314), ('HouseAge', 0.04361683830030022), ('AveRooms', 0.020705699971547562), ('AveBedrms', 0.008529440735645566), ('Population', 0.0)]
rf = RandomForestRegressor(seed=0)
rf_model = rf.fit(train2)
print(sorted(zip(california.feature_names, rf_model.featureImportances), key=lambda tup: -tup[1]))
#Before Change: [('MedInc', 0.5960043801299608), ('AveOccup', 0.11802695085456516), ('Latitude', 0.10557829783042827), ('AveRooms', 0.08226198073251881), ('Longitude', 0.05014360511503636), ('HouseAge', 0.036513356705612766), ('AveBedrms', 0.01075420375328864), ('Population', 0.0007172248785891252)]
#After Change: [('MedInc', 0.5960043801299608), ('AveOccup', 0.11802695085456516), ('Latitude', 0.10557829783042827), ('AveRooms', 0.08226198073251881), ('Longitude', 0.05014360511503636), ('HouseAge', 0.036513356705612766), ('AveBedrms', 0.01075420375328864), ('Population', 0.0007172248785891252)]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org