CGBoost: Conjugate Gradient in Function Space

Li, Ling and Abu-Mostafa, Yaser S. and Pratap, Amrit (2003) CGBoost: Conjugate Gradient in Function Space. Technical Report. California Institute of Technology. [CaltechCSTR:2003.007]

Full text available as:

Postscript - Requires a viewer, such as GhostView

Abstract

The superior out-of-sample performance of AdaBoost has been attributed to the fact that it minimizes a cost function based on margin, in that it can be viewed as a special case of AnyBoost, an abstract gradient descent algorithm. In this paper, we provide a more sophisticated abstract boosting algorithm, CGBoost, based on conjugate gradient in function space. When the AdaBoost exponential cost function is optimized, CGBoost generally yields much lower cost and training error but higher test error, which implies that the exponential cost is vulnerable to overfitting. With the optimization power of CGBoost, we can adopt more "regularized" cost functions that have better out-of-sample performance but are difficult to optimize. Our experiments demonstrate that CGBoost generally outperforms AnyBoost in cost reduction. With suitable cost functions, CGBoost can have better out-of-sample performance.

EPrint Type:	Monograph (Technical Report)
Additional Information:	[Alternate URL: http://www.cs.caltech.edu/~ling/pub/tr03cgboost.pdf]
Uncontrolled Keywords:	conjugate-gradient, cost function, boosting, generalization, AdaBoost, AnyBoost, CGBoost
Subjects:	All Records
ID Code:	500
Deposited By:	Ling Li
Deposited On:	29 August 2003
Record Number:	CaltechCSTR:2003.007
Official Persistent URL:	http://resolver.caltech.edu/CaltechCSTR:2003.007
Usage Policy:	You are granted permission for individual, educational, research and non-commercial reproduction, distribution, display and performance of this work in any format.

Archive Staff Only: edit this record