Benchmarks for Optimization Software

By Hans Mittelmann (mittelmann at asu.edu)


END OF A BENCHMARKING ERA
 For many years our benchmarking effort had included the solvers CPLEX, Gurobi, and XPRESS. Through an action by Gurobi at the 2018 INFORMS Annual Meeting this has come to an end. IBM and FICO demanded that results for their solvers be removed and then we decided to temporarily remove those of Gurobi as well. CPLEX was in fifteen of the benchmarks, Gurobi and XPRESS in thirteen. See here for more details. In late November 2019 selected benchmarks for Gurobi were added.
A partial record of previous benchmarks can be obtained from this webpage and some additional older benchmarks

Note that on top of the benchmarks a link to logfiles is given!

See this graphical tool for visualization of most of the results including a virtual best (ensemble)

Note also that we do not use performance profiles. See this paper and that one

We use instead the shifted geometric mean


COMBINATORIAL OPTIMIZATION

Concorde-TSP with different LP solvers (3-3-2023)


LINEAR PROGRAMMING

The Simplex and Barrier benchmarks are replaced by benchmarks that show how well solvers find a primal-dual feasible point (as barrier methods in general do) or an optimal basic solution (as simplex methods in general do). Codes do not have to use a textbook version of either method.

LPfeas Benchmark (find a PD feasible point) (3-24-2023)

LPopt Benchmark (find optimal basic solution) (3-24-2023)

Large Network-LP Benchmark (commercial vs free) (3-24-2023)


MIXED INTEGER LINEAR PROGRAMMING

MILP Benchmark - MIPLIB2017 (4-1-2023)

MILP cases that are slightly pathological (4-9-2023)

Infeasibility Detection for MILP Problems (4-13-2023)


SEMIDEFINITE/SQL PROGRAMMING

SQL problems from the 7th DIMACS Challenge (8-8-2002)

Several SDP codes on sparse and other SDP problems (3-24-2023)

Infeasible SDP Benchmark (3-24-2023)

Large SOCP Benchmark (3-28-2023)

MISOCP Benchmark (3-25-2023)


NONLINEAR PROGRAMMING

AMPL-NLP Benchmark (1-18-2022)


MIXED INTEGER QPs and QCPs

Non-commercial convex QP Benchmark (9-16-2021)

Binary Non-Convex QPLIB Benchmark (4-20-2023)

Non-Convex QUBO-QPLIB Benchmark (4-22-2023)

Discrete Non-Convex QPLIB Benchmark (non-binary) (4-23-2023)

Continuous Non-Convex QPLIB Benchmark (4-26-2023)

Convex Continuous QPLIB Benchmark (5-6-2023)

Convex Discrete QPLIB Benchmark (4-19-2023)


MIXED INTEGER NONLINEAR PROGRAMMING

MINLP Benchmark (4-20-2023)


PROBLEMS WITH EQUILIBRIUM CONSTRAINTS

MPEC Benchmark (4-12-2022)