A Case Study on Bagging, Boosting, and Basic Ensembles of Neural Networks for OCR

A Case Study on Bagging, Boosting, and Basic Ensembles of Neural

Networks for OCR

Jianchang Mao

IBM Almaden Research Center

650 Harry Road

San Jose, California 95120-6099

mao@almaden . i bm. com

Abstract

We study the effectiveness of three neural network

ensembles in improving OCR performance: (i) Ba- sic, (ii) Bagging, and (iii) Boosting. Three random character degradation models are introduced in train-

ing indivadual networks in order to reduce error cor-

relation between individual networks and to improve

the generalization ability of neural networks. We com-

pare the recognition accuracies of these three ensem-

bles at various reject rates. An interesting discovery in

our comparison is that although the Boosting ensem-

ble is slightly more accurate than the Basic ensemble and Bagging ensemble at zero reject rate, the advan-

tage of the Boosting training over the Basic and Bag-

gang ensembles quickly disappears as more patterns are

rejected, Eventually the Basic and Bagging ensembles

outperform the Boosting ensemble at high reject rates.

Explanation of such a phenomenon is provided in the

paper. We also apply the optimal linear combiner (in

the least square error sense) to each of the three en-

sembles to capture dafferent error correlatzon charac-

teristics of the three ensembles. We find that the opti- mal linear combiner is very effective in reducang mean

square error, but is not necessarily as effective as a

simply average method in reducing classijcation error.

1 Introduction

Combining multiple classifiers has received a consid-

erable amount of interest in recent years [20, 7, lo]. It is believed to be an effective way to design classifiers

with high recognition accuracy, reliability, and robust-

ness, to deal with the variance-bias dilemma [18, 21, to handle different feature types and scales, and to maxi- mally exploit the discriminant power of individual fea-

tures and classifiers (experts).

It is generally believed that a good ensemble should consist of a set of individual classifiers which are indi-

vidually accurate and are loosely correlated in making errors [6, 111. Various techniques have been introduced to produce such classifiers. For example, one can em-

ploy different types of classifiers and different feature

representations. One can also train individual classi-

fiers on different sample sets drawn from the original training data.

Bagging [l] and Boosting [16, 4, 51 are two repre- sentative methods for creating an ensemble of classi-

fiers based on statistical re-sampling techniques. Previ- ous studies have shown that these two methods can ef- fectively improve recognition accuracy over individual

classifiers such as decision trees [l, 2, 3, 151 and neural

networks [4, 131. A number of comparative studies of

Bagging and Boosting [13, 2, 151 have been reported

in the literature. Although there is no strong conclu-

sion on which method is superior to the other, sev- eral general observations have been made [13, 151: (i)

The improvement on recognition accuracy from Boost-

ing has a large variance. In some cases, Boosting can

significantly outperform Bagging, while in some other

cases, it can also be substantially worse than Bagging

(in a few cases even worse than individual classifiers); (ii) Bagging’s improvement over individual classifiers is more consistent on various data sets than Boosting’s.

All these comparative studies concern only the

recognition accuracy without any rejection. In many real-world applications, such as bank check reading and fingerprint identification, any substitution error could be very expensive. In these cases, we have to reduce the

substitution error rate by rejecting unreliable recogni-

tions. Therefore, a comparison of ensembles at various reject rates becomes necessary.

0-7803-4859- 1/98 $ I0.0001998 IEEE 1828

In this paper, we study the effectiveness of three

neural network ensembles (Basic, Bagging, and Boost-

ing) in improving OCR performance. The performance

is evaluated wiith or without rejection separately. An interesting observation which has not been made in pre- vious studies aril1 be provided. Three random charac- ter degradation models are introduced in training in-

dividual networks to reduce error correlation between individual networks and to improve their generaliza-

tion behavior. The characteristics of the error corre-

lation between the individual networks in each of the

three ensembles are analyzed. In order to make iise of

these different error characteristics, the optimal linear

combiner (generalized ensemble method or GEM [14]) is used to combine outputs of individual netwoIks in each of the three ensemble. The performance of GEM is compared with the averaging method