Accelerated proximal boosting

Erwan Fouillen (1) , Claire Boyer (1, 2) , Maxime Sangnier (1, 3)

1 LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation
2 DMA - Département de Mathématiques et Applications - ENS Paris
3 MLIA - Machine Learning and Information Access

Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm when the empirical risk to minimize is not differentiable. In addition , the novel boosting approach, called accelerated proximal boosting, benefits from Nesterov's acceleration in the same way as gradient boosting [Biau et al., 2018]. Advantages of leveraging proximal methods for boosting are illustrated by numerical experiments on simulated and real-world data. In particular, we exhibit a favorable comparison over gradient boosting regarding convergence rate and prediction accuracy.

Domaines: Machine Learning [stat.ML]; Apprentissage [cs.LG]

Erwan Fouillen, Claire Boyer, Maxime Sangnier. Accelerated proximal boosting. 2018. ⟨hal-01853244v1⟩