Ridge-Adjusted Slack Variable Optimization for Supervised Classification

Paper in proceedings, 2013

This paper presents an iterative classification algorithm called Ridge-adjusted Slack Variable Optimization (RiSVO). RiSVO is an iterative procedure with two steps: (1) A working subset of the training data is selected so as to reject "extreme" patterns. (2) the decision vector and threshold value are obtained by minimizing the energy function associated with the slack variables. From a computational perspective, we have established a sufficient condition for the "inclusion property" among successive working sets, which allows us to save computation time. Most importantly, under the inclusion property, the monotonic reduction of the energy function can be assured in both substeps at each iteration, thus assuring the convergence of the algorithm. Moreover, ridge regularization is incorporated to improve the robustness and better cope with over-fitting and ill-conditioned problems. To verify the proposed algorithm, we conducted simulations on three data sets from the UCI database: adult, shuttle and bank. Our simulation shows stability and convergence of the RiSVO method. The results also show improvement of performance over the SVM classifier.