Statistical Modelling 15 (5) (2015), 389–410

Selection and Fusion of Categorical Predictors with L0-Type Penalties

Margret-Ruth Oelker
Department of Statistics,
Ludwig-Maximilians-Universität München,
Germany
e-mail: margret.oelker@stat.uni-muenchen.de

Wolfgang Pößnecker
Department of Statistics,
Ludwig-Maximilians-Universität München,
Germany


Gerhard Tutz
Department of Statistics,
Ludwig-Maximilians-Universität München,
Germany


Abstract:

In regression modelling, categorical covariates have to be coded. Depending on the number of categorical covariates and on the number of levels they have, the number of coefficients can become huge. To reduce the model complexity, coefficients of similar categories should be fused and coefficients of non-influential categories should be set to zero. To this end, Lasso-type penalties on the differences of coefficients are a standard approach. However, the clustering/selection performance of this approach is sometimes poor–especially when the adaptive weights are badly conditioned or not existing. In some situations, there is no incentive to cluster similar categories. To overcome this, a L0 penalty on the differences of coefficients is proposed, whereby the L0 ‘norm’ is defined as the number of non-zero entries in a vector. The proposed penalty favours to find clusters of categories that share the same effect on the response variable while the estimation accuracy is comparable to Lasso-type penalties. Numerical experiments within the framework of generalized linear models are promising. For illustration, data on the unemployment rates in Germany is analyzed.

Keywords:

adaptive Lasso; best subset selection; GLMs; model selection.

Downloads:

Example data, code and supplement material in zipped archive.
back