OrdinalCategoricalBucketer

Bases: BaseBucketer

The OrdinalCategoricalBucketer replaces categories by ordinal numbers.

Support

When sort_by_target is false the buckets are assigned in order of frequency. When sort_by_target is true the buckets are ordered based on the mean of the target per category.

For example, if for a variable colour the means of the target for blue, red and grey is 0.5, 0.8 and 0.1 respectively, grey will be the first bucket (0), blue the second (1) and red the third (3). If new data contains unknown labels (f.e. yellow), they will be replaced by the 'Other' bucket (-2), and if new data contains missing values, they will be replaced by the 'Missing' bucket (-1).

Example:

from skorecard import datasets
from skorecard.bucketers import OrdinalCategoricalBucketer

X, y = datasets.load_uci_credit_card(return_X_y=True)
bucketer = OrdinalCategoricalBucketer(variables=['EDUCATION'])
bucketer.fit_transform(X, y)
bucketer = OrdinalCategoricalBucketer(max_n_categories=2, variables=['EDUCATION'])
bucketer.fit_transform(X, y)

Credits: Code & ideas adapted from:

• feature_engine.categorical_encoders.OrdinalCategoricalEncoder
• feature_engine.categorical_encoders.RareLabelCategoricalEncoder

Source code in skorecard/bucketers/bucketers.py

class OrdinalCategoricalBucketer(BaseBucketer):
    """
    The `OrdinalCategoricalBucketer` replaces categories by ordinal numbers.

    Support ![badge](https://img.shields.io/badge/numerical-false-red) ![badge](https://img.shields.io/badge/categorical-true-green) ![badge](https://img.shields.io/badge/supervised-true-green)

    When `sort_by_target` is `false` the buckets are assigned in order of frequency.
    When `sort_by_target` is `true` the buckets are ordered based on the mean of the target per category.

    For example, if for a variable `colour` the means of the target
    for `blue`, `red` and `grey` is `0.5`, `0.8` and `0.1` respectively,
    `grey` will be the first bucket (`0`), blue the second (`1`) and
    `red` the third (`3`). If new data contains unknown labels (f.e. yellow),
    they will be replaced by the 'Other' bucket (`-2`),
    and if new data contains missing values, they will be replaced by the 'Missing' bucket (`-1`).

    Example:

    ```python
    from skorecard import datasets
    from skorecard.bucketers import OrdinalCategoricalBucketer

    X, y = datasets.load_uci_credit_card(return_X_y=True)
    bucketer = OrdinalCategoricalBucketer(variables=['EDUCATION'])
    bucketer.fit_transform(X, y)
    bucketer = OrdinalCategoricalBucketer(max_n_categories=2, variables=['EDUCATION'])
    bucketer.fit_transform(X, y)
    ```

    Credits: Code & ideas adapted from:

    - feature_engine.categorical_encoders.OrdinalCategoricalEncoder
    - feature_engine.categorical_encoders.RareLabelCategoricalEncoder

    """  # noqa

    def __init__(
        self,
        tol=0.05,
        max_n_categories=None,
        variables=[],
        specials={},
        encoding_method="frequency",
        missing_treatment="separate",
        remainder="passthrough",
        get_statistics=True,
    ):
        """
        Init the class.

        Args:
            tol (float): the minimum frequency a label should have to be considered frequent.
                Categories with frequencies lower than tol will be grouped together (in the 'other' bucket).
            max_n_categories (int): the maximum number of categories that should be considered frequent.
                If None, all categories with frequency above the tolerance (tol) will be
                considered.
            variables (list): The features to bucket. Uses all features if not defined.
            specials (dict): (nested) dictionary of special values that require their own binning.
                The dictionary has the following format:
                 {"<column name>" : {"name of special bucket" : <list with 1 or more values>}}
                For every feature that needs a special value, a dictionary must be passed as value.
                This dictionary contains a name of a bucket (key) and an array of unique values that should be put
                in that bucket.
                When special values are defined, they are not considered in the fitting procedure.
            encoding_method (string): encoding method.
                - "frequency" (default): orders the buckets based on the frequency of observations in the bucket.
                    The lower the number of the bucket the most frequent are the observations in that bucket.
                - "ordered": orders the buckets based on the average class 1 rate in the bucket.
                    The lower the number of the bucket the lower the fraction of class 1 in that bucket.
            missing_treatment (str or dict): Defines how we treat the missing values present in the data.
                If a string, it must be one of the following options:
                    separate: Missing values get put in a separate 'Other' bucket: `-1`
                    most_risky: Missing values are put into the bucket containing the largest percentage of Class 1.
                    least_risky: Missing values are put into the bucket containing the largest percentage of Class 0.
                    most_frequent: Missing values are put into the most common bucket.
                    neutral: Missing values are put into the bucket with WoE closest to 0.
                    similar: Missing values are put into the bucket with WoE closest to the bucket with only missing values.
                    passthrough: Leaves missing values untouched.
                If a dict, it must be of the following format:
                    {"<column name>": <bucket_number>}
                    This bucket number is where we will put the missing values.
            remainder (str): How we want the non-specified columns to be transformed. It must be in ["passthrough", "drop"].
                passthrough (Default): all columns that were not specified in "variables" will be passed through.
                drop: all remaining columns that were not specified in "variables" will be dropped.
        """  # noqa
        self.tol = tol
        self.max_n_categories = max_n_categories
        self.variables = variables
        self.specials = specials
        self.encoding_method = encoding_method
        self.missing_treatment = missing_treatment
        self.remainder = remainder
        self.get_statistics = get_statistics

    @property
    def variables_type(self):
        """
        Signals variables type supported by this bucketer.
        """
        return "categorical"

    def _get_feature_splits(self, feature, X, y, X_unfiltered=None):
        """
        Finds the splits for a single feature.

        X and y have already been preprocessed, and have specials removed.

        Args:
            feature (str): Name of the feature.
            X (pd.Series): df with single column of feature to bucket
            y (np.ndarray): array with target
            X_unfiltered (pd.Series): df with single column of feature to bucket before any filtering was applied

        Returns:
            splits, right (tuple): The splits (dict or array), and whether right=True or False.
        """
        normalized_counts = None

        if y is None:
            y = pd.Series(None)
        elif not (isinstance(y, pd.Series) or isinstance(y, pd.DataFrame)):
            y = pd.Series(y)
        else:
            raise AssertionError("something wrong with format of y")

        X_y = pd.concat([X, y], axis=1)
        X_y.columns = [feature, "target"]

        if self.encoding_method == "ordered":
            if y is None:
                raise ValueError("To use encoding_method=='ordered', y cannot be None.")

            normalized_counts = X_y[feature].value_counts(normalize=True)
            cats = X_y.groupby([feature])["target"].mean().sort_values(ascending=True).index
            normalized_counts = normalized_counts[cats]

        elif self.encoding_method == "frequency":
            normalized_counts = X_y[feature].value_counts(normalize=True)

        # Limit number of categories if set.
        normalized_counts = normalized_counts[: self.max_n_categories]
        # Remove less frequent categories
        normalized_counts = normalized_counts[normalized_counts >= self.tol]

        # Determine Ordinal Encoder based on ordered labels
        # Note we start at 1, to be able to encode missings as 0.
        mapping = dict(zip(normalized_counts.index, range(0, len(normalized_counts))))

        # Note that right is set to True, but this is not used at all for categoricals
        return (mapping, True)

variables_type property

Signals variables type supported by this bucketer.

__init__(tol=0.05, max_n_categories=None, variables=[], specials={}, encoding_method='frequency', missing_treatment='separate', remainder='passthrough', get_statistics=True)

Init the class.

Parameters:

Name	Type	Description	Default
tol	float	the minimum frequency a label should have to be considered frequent. Categories with frequencies lower than tol will be grouped together (in the 'other' bucket).	0.05
max_n_categories	int	the maximum number of categories that should be considered frequent. If None, all categories with frequency above the tolerance (tol) will be considered.	None
variables	list	The features to bucket. Uses all features if not defined.	[]
specials	dict	(nested) dictionary of special values that require their own binning. The dictionary has the following format: {"" : {"name of special bucket" : }} For every feature that needs a special value, a dictionary must be passed as value. This dictionary contains a name of a bucket (key) and an array of unique values that should be put in that bucket. When special values are defined, they are not considered in the fitting procedure.	{}
encoding_method	string	encoding method. - "frequency" (default): orders the buckets based on the frequency of observations in the bucket. The lower the number of the bucket the most frequent are the observations in that bucket. - "ordered": orders the buckets based on the average class 1 rate in the bucket. The lower the number of the bucket the lower the fraction of class 1 in that bucket.	'frequency'
missing_treatment	str or dict	Defines how we treat the missing values present in the data. If a string, it must be one of the following options: separate: Missing values get put in a separate 'Other' bucket: -1 most_risky: Missing values are put into the bucket containing the largest percentage of Class 1. least_risky: Missing values are put into the bucket containing the largest percentage of Class 0. most_frequent: Missing values are put into the most common bucket. neutral: Missing values are put into the bucket with WoE closest to 0. similar: Missing values are put into the bucket with WoE closest to the bucket with only missing values. passthrough: Leaves missing values untouched. If a dict, it must be of the following format: {"": } This bucket number is where we will put the missing values.	'separate'
remainder	str	How we want the non-specified columns to be transformed. It must be in ["passthrough", "drop"]. passthrough (Default): all columns that were not specified in "variables" will be passed through. drop: all remaining columns that were not specified in "variables" will be dropped.	'passthrough'

Source code in skorecard/bucketers/bucketers.py

def __init__(
    self,
    tol=0.05,
    max_n_categories=None,
    variables=[],
    specials={},
    encoding_method="frequency",
    missing_treatment="separate",
    remainder="passthrough",
    get_statistics=True,
):
    """
    Init the class.

    Args:
        tol (float): the minimum frequency a label should have to be considered frequent.
            Categories with frequencies lower than tol will be grouped together (in the 'other' bucket).
        max_n_categories (int): the maximum number of categories that should be considered frequent.
            If None, all categories with frequency above the tolerance (tol) will be
            considered.
        variables (list): The features to bucket. Uses all features if not defined.
        specials (dict): (nested) dictionary of special values that require their own binning.
            The dictionary has the following format:
             {"<column name>" : {"name of special bucket" : <list with 1 or more values>}}
            For every feature that needs a special value, a dictionary must be passed as value.
            This dictionary contains a name of a bucket (key) and an array of unique values that should be put
            in that bucket.
            When special values are defined, they are not considered in the fitting procedure.
        encoding_method (string): encoding method.
            - "frequency" (default): orders the buckets based on the frequency of observations in the bucket.
                The lower the number of the bucket the most frequent are the observations in that bucket.
            - "ordered": orders the buckets based on the average class 1 rate in the bucket.
                The lower the number of the bucket the lower the fraction of class 1 in that bucket.
        missing_treatment (str or dict): Defines how we treat the missing values present in the data.
            If a string, it must be one of the following options:
                separate: Missing values get put in a separate 'Other' bucket: `-1`
                most_risky: Missing values are put into the bucket containing the largest percentage of Class 1.
                least_risky: Missing values are put into the bucket containing the largest percentage of Class 0.
                most_frequent: Missing values are put into the most common bucket.
                neutral: Missing values are put into the bucket with WoE closest to 0.
                similar: Missing values are put into the bucket with WoE closest to the bucket with only missing values.
                passthrough: Leaves missing values untouched.
            If a dict, it must be of the following format:
                {"<column name>": <bucket_number>}
                This bucket number is where we will put the missing values.
        remainder (str): How we want the non-specified columns to be transformed. It must be in ["passthrough", "drop"].
            passthrough (Default): all columns that were not specified in "variables" will be passed through.
            drop: all remaining columns that were not specified in "variables" will be dropped.
    """  # noqa
    self.tol = tol
    self.max_n_categories = max_n_categories
    self.variables = variables
    self.specials = specials
    self.encoding_method = encoding_method
    self.missing_treatment = missing_treatment
    self.remainder = remainder
    self.get_statistics = get_statistics