Automatic Feature selection techniques¶

Below we'll walk you through how to change selecting the best feature_set manually to automatically by using get_reduced_features_set. This function has many ways of selecting the most optimal feature selection for your use-case. Let's show you first how a simple 'manaul' probatus feature elimination works before we'll show you how to automate it.

In [ ]:

%%capture
!pip install probatus
!pip install catboost

%%capture !pip install probatus !pip install catboost

In [1]:

from catboost import CatBoostClassifier
from sklearn.datasets import make_classification

from probatus.feature_elimination import ShapRFECV

from catboost import CatBoostClassifier from sklearn.datasets import make_classification from probatus.feature_elimination import ShapRFECV

/home/bibimb/projects/probatus/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Basic example of ShapRFECV¶

In [2]:

# Simple ShapRFECV example
X, y = make_classification(n_samples=500, n_informative=20, n_features=50)
model = CatBoostClassifier(n_estimators=100, verbose=0)
shap_elimination = ShapRFECV(model, step=0.2, min_features_to_select=5, cv=5, scoring="f1")
report = shap_elimination.fit_compute(X, y)

# Simple ShapRFECV example X, y = make_classification(n_samples=500, n_informative=20, n_features=50) model = CatBoostClassifier(n_estimators=100, verbose=0) shap_elimination = ShapRFECV(model, step=0.2, min_features_to_select=5, cv=5, scoring="f1") report = shap_elimination.fit_compute(X, y)

In [3]:

# Inspect the report generated by ShapRFECV
# Can you find the best iteration? Look at `train_metric_mean`.
report

# Inspect the report generated by ShapRFECV # Can you find the best iteration? Look at `train_metric_mean`. report

Out[3]:

	num_features	features_set	eliminated_features	train_metric_mean	train_metric_std	val_metric_mean	val_metric_std
1	50	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,...	[38, 4, 31, 1, 18, 49, 34, 5, 36, 9]	1.000000	0.000000	0.772154	0.029291
2	40	[0, 2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16,...	[7, 30, 25, 16, 32, 15, 48, 0]	1.000000	0.000000	0.784704	0.036232
3	32	[2, 3, 6, 8, 10, 11, 12, 13, 14, 17, 19, 20, 2...	[14, 2, 20, 47, 37, 35]	1.000000	0.000000	0.808936	0.018636
4	26	[3, 6, 8, 10, 11, 12, 13, 17, 19, 21, 22, 23, ...	[27, 43, 42, 11, 45]	1.000000	0.000000	0.816919	0.013187
5	21	[3, 6, 8, 10, 12, 13, 17, 19, 21, 22, 23, 24, ...	[8, 44, 33, 6]	0.999494	0.001013	0.814677	0.023937
6	17	[3, 10, 12, 13, 17, 19, 21, 22, 23, 24, 26, 28...	[10, 17, 22]	0.951183	0.005158	0.797702	0.037003
7	14	[3, 12, 13, 19, 21, 23, 24, 26, 28, 29, 39, 40...	[28, 19]	0.938864	0.003603	0.795212	0.029251
8	12	[3, 12, 13, 21, 23, 24, 26, 29, 39, 40, 41, 46]	[29, 23]	0.940406	0.004328	0.789350	0.050198
9	10	[3, 12, 13, 21, 24, 26, 39, 40, 41, 46]	[3, 26]	0.925328	0.004082	0.782151	0.054552
10	8	[12, 13, 21, 24, 39, 40, 41, 46]	[40]	0.901369	0.002607	0.779374	0.029862
11	7	[12, 13, 21, 24, 39, 41, 46]	[13]	0.881766	0.008313	0.764759	0.033718
12	6	[12, 21, 24, 39, 41, 46]	[12]	0.857728	0.006382	0.764569	0.027978
13	5	[21, 24, 39, 41, 46]	[]	0.841456	0.004388	0.745611	0.036180

In [4]:

# We can visually inspect the report and try identify the best iteration
shap_elimination.plot()

# We can visually inspect the report and try identify the best iteration shap_elimination.plot()

Out[4]:

In [5]:

# Once we have identified which iteration was best, we can get the feature names by:
shap_elimination.get_reduced_features_set(num_features=21)

# Once we have identified which iteration was best, we can get the feature names by: shap_elimination.get_reduced_features_set(num_features=21)

Out[5]:

[3,
 6,
 8,
 10,
 12,
 13,
 17,
 19,
 21,
 22,
 23,
 24,
 26,
 28,
 29,
 33,
 39,
 40,
 41,
 44,
 46]

But, is there a more automated way of selecting "best" features?¶

Yes, probatus support a few strategies for selecting the best num_features for your needs.

Before we jump into these automated techniques, there are a number of factors you might want to consider when selecting the best iteration:

1. Model performance score: Most people will care about this the most. You'll try to maximize a score like f1 or accuracy or ROC_AUC. But there are more factors to consider...
2. Variance of model performance: If your model performance scores vary considerably across cross validation folds, it might be a sign that your model is not generalizing well. Some people may want to ensure model performance standard deviation is also minimized.
3. Number of features: You might want to keep the number of features to a minimum (smaller model, siplier data pipeline, easier to explain model)

To accomodate some of the above needs, probatus supports automated strategies to select the best "num_features". Strategies supported are:

1. num_features="best": If what you care about is maximizing val_metric_mean (model performance score), this strategy is for you.
2. num_features="best_coherent": If you care about high model performance whilst also minimizing variation of model performance between cross validation folds, this strategy is for you.
3. num_features="best_parsimonious": If you care about high model performance whilst also minimizing the number of features selected, this model is for you.

best_coherent and best_parsimonious consider only iterations whose mean validation score is at least best_mean - standard_error_threshold * best_std. Here best_std is the standard deviation across validation folds of the iteration with the highest mean score. Despite the historical parameter name, this is a standard-deviation multiplier, not a standard error of the mean or an absolute score tolerance. The default multiplier is 1.0; zero requires a score equal to the best mean. Among eligible iterations, best_coherent chooses the lowest score standard deviation and best_parsimonious the fewest features.

In [6]:

# Best
best_features = shap_elimination.get_reduced_features_set(num_features="best")
print(f"The {len(best_features)} best features are: {best_features}")

# Best best_features = shap_elimination.get_reduced_features_set(num_features="best") print(f"The {len(best_features)} best features are: {best_features}")

The 26 best features are: [3, 6, 8, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 33, 39, 40, 41, 42, 43, 44, 45, 46]

In [7]:

# Best coherent
best_features = shap_elimination.get_reduced_features_set(num_features="best_coherent")
print(f"The {len(best_features)} best coherent features are: {best_features}")

# Best coherent best_features = shap_elimination.get_reduced_features_set(num_features="best_coherent") print(f"The {len(best_features)} best coherent features are: {best_features}")

The 26 best coherent features are: [3, 6, 8, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 33, 39, 40, 41, 42, 43, 44, 45, 46]

In [8]:

# Best parsimonious
best_features = shap_elimination.get_reduced_features_set(num_features="best_parsimonious")
print(f"The {len(best_features)} best parsimonious features are: {best_features}")

# Best parsimonious best_features = shap_elimination.get_reduced_features_set(num_features="best_parsimonious") print(f"The {len(best_features)} best parsimonious features are: {best_features}")

The 21 best parsimonious features are: [3, 6, 8, 10, 12, 13, 17, 19, 21, 22, 23, 24, 26, 28, 29, 33, 39, 40, 41, 44, 46]

In [9]:

# Note, you can change the behavior of `best_coherent` and `best_parsimonious` by changing the `standard_error_threshold` parameter
# Best coherent (standard_error_threshold=0.5)
best_features = shap_elimination.get_reduced_features_set(num_features="best_coherent", standard_error_threshold=0.5)
print(f"The {len(best_features)} best coherent features are: {best_features}")

# Note, you can change the behavior of `best_coherent` and `best_parsimonious` by changing the `standard_error_threshold` parameter # Best coherent (standard_error_threshold=0.5) best_features = shap_elimination.get_reduced_features_set(num_features="best_coherent", standard_error_threshold=0.5) print(f"The {len(best_features)} best coherent features are: {best_features}")

The 26 best coherent features are: [3, 6, 8, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 33, 39, 40, 41, 42, 43, 44, 45, 46]

Support & Ranking¶

In alignment with the sklearn implementation of RFECV, you can get support and ranking output from ShapRFECV.

In [10]:

# Get support (boolean mask of X feature columns)
best_features = shap_elimination.get_reduced_features_set(num_features="best", return_type="feature_names")
best_features_support = shap_elimination.get_reduced_features_set(num_features="best", return_type="support")
best_features_ranking = shap_elimination.get_reduced_features_set(num_features="best", return_type="ranking")
print(f"The {len(best_features)} best features are: {best_features}")
print(f"Support boolean mask: {best_features_support}")
print(f"Ranking: {best_features_ranking}")

# Get support (boolean mask of X feature columns) best_features = shap_elimination.get_reduced_features_set(num_features="best", return_type="feature_names") best_features_support = shap_elimination.get_reduced_features_set(num_features="best", return_type="support") best_features_ranking = shap_elimination.get_reduced_features_set(num_features="best", return_type="ranking") print(f"The {len(best_features)} best features are: {best_features}") print(f"Support boolean mask: {best_features_support}") print(f"Ranking: {best_features_ranking}")

The 26 best features are: [3, 6, 8, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 33, 39, 40, 41, 42, 43, 44, 45, 46]
Support boolean mask: [False, False, False, True, False, False, True, False, True, False, True, True, True, True, False, False, False, True, False, True, False, True, True, True, True, False, True, True, True, True, False, False, False, True, False, False, False, False, False, True, True, True, True, True, True, True, True, False, False, False]
Ranking: [35, 39, 23, 4, 37, 43, 16, 28, 13, 45, 10, 20, 1, 2, 22, 33, 31, 11, 40, 9, 24, 0, 12, 7, 0, 30, 5, 17, 8, 6, 29, 38, 32, 15, 42, 27, 44, 26, 36, 0, 3, 0, 19, 18, 14, 21, 0, 25, 34, 41]