Optimization Online


A Statistical Test for Comparing Success Rates

Eric Taillard (eric.taillard***at***eivd.ch)

Abstract: This article presents a non-parametric statistical test that is very interesting for those who want to compare different heuristic algorithms that do not necessarily end with feasible (or satisfying) solutions. This test has been specially designed for working with very small sample sizes, meaning that a substantial computational effort can be saved when conducting numerical experiments. When the sample sizes are lower than 15, standard statistical tests for comparing the success rates of two populations cannot be validly used. So it is for very high confidence rates, even if sample sizes are larger than 15. Therefore, a non parametric test has been developed. This test is more powerful than Mc Nemar's one and can be applied for any sample sizes, but it requires relatively heavy computations. So, pre-computed values for 95% and 99% confidence levels have been tabulated in the article. The computation of confidence levels can also be done online at the URL: http://ina.eivd.ch/projects/stamp/

Keywords: Statistical test, comparison of heuristics

Category 1: Applications -- OR and Management Sciences

Citation: Metaheuristic international conference MIC'03, Kyoto, Japan, August, 2003

Download: [PDF]

Entry Submitted: 11/28/2003
Entry Accepted: 11/30/2003
Entry Last Modified: 11/28/2003

Modify/Update this entry

  Visitors Authors More about us Links
  Subscribe, Unsubscribe
Digest Archive
Search, Browse the Repository


Coordinator's Board
Classification Scheme
Give us feedback
Optimization Journals, Sites, Societies
Mathematical Programming Society