“Experiments With Highleyman’s Data”, 1968-04-01 ():
The results of 3 experiments with Highleyman’s hand-printed characters are reported.
Nearest-neighbor classification is used to explain the high error rates (42 to 60%) obtained by general statistical procedures. An error rate of 32% is obtained by preceding piecewise-linear classification by edge-detecting preprocessing.
The minimum human error rate is estimated, and suggested as a performance standard.
[Keywords: character recognition, classification, feature extraction, human performance, nearest-neighbor classification, pattern recognition, preprocessing]
…Most subjects were satisfied with the training phase after they had seen 75 to 100 characters, and volunteered to move on to the testing phase. On the test data, their error rates ranged 13.6%–18.3%, with an average error rate of 15.7%. Assuming a normal distribution of scores, this indicates that, with 95% confidence, the true mean error rate is 15.7% ±0.9%.
These numbers include a fair proportion of errors due to confusions between I and 1 and O and 0. If these errors are not counted, the mean error rate drops to 11.5%, which is still considerably greater than the 4.1% reported by Neisser & Weene for their unquantized characters. If the I-1 and O-0 distinctions are retained, but if a plurality vote of the 10 separate responses is used to classify the characters (ties being broken arbitrarily), then an error rate of 11.4% results. We believe that this value is close to the minimum error rate that can be achieved with Highleyman’s data, and that the performance of other methods on the 36-character test data should be viewed relative to this standard.
…While the development of more effective preprocessing and classification techniques for Highleyman’s data may be a challenging problem in itself, we feel that larger and higher-quality data sets are needed for work aimed at achieving useful results. Such datasets may contain hundreds, or even thousands, of samples in each class. We know, for example, that investigators at SRI and IBM have used data sets containing over 10 thousand samples, and we expect that even larger data sets will be collected. Experience with such data suggests that an array size of at least 20×20 is needed, with an optimum size of perhaps 30×30.
View PDF: