The Simplex Method for solving real-valued linear optimalization problems

Simplex method: introduction

From Wikipedia:

The simplex algorithm (or simplex method), created by the American mathematician George Dantzig in 1947, is a very popular algorithm for solving linear programs.
The journal Computing in Science and Engineering listed it as one of the top 10 algorithms of the twentieth century.
Wikipedia page: click here
The name of the algorithm is derived from the concept of a simplex and was suggested by T. S. Motzkin.
Wikipedia page: click here

Presentation:

There is a rich Mathematical background to the Simplex Method...
For brevity, I will show you what to do
I will not explain why it works....
(It will take too much time to explain, and you may not have the background - heavy duty Linear Algebra)

The standard LP

The standard LP has the following form:

max: a₁x₁ + a₂x₂ + ... + a_nx_n // Objective function subject to: c₁₁x₁ + c₁₂x₂ + ... + c_1nx_n ≤ d₁ // Constraints c₂₁x₁ + c₂₂x₂ + ... + c_2nx_n ≤ d₂ .... c_m1x₁ + c_m2x₂ + ... + c_mnx_n ≤ d_m x₁ ≥ 0, x₂ ≥ 0, ..., x_n ≥ 0

Note:

The standard LP problem uses only "≤"-constraints
The reason for this is:

Fact:
We must first learn how to convert the standard LP into a "matrix" (tableau) form !!!

I I I I

Prelude to conversion to Simplex Tableau: converting "≤"-constraints into "="-constraints

Fact:

Matrix row operations operates on linear equations of the following form:

c₁₁x₁ + c₁₂x₂ + ... + c_1nx_n = d₁ c₂₁x₁ + c₂₂x₂ + ... + c_2nx_n = d₂ .... c_m1x₁ + c_m2x₂ + ... + c_mnx_n = d_m

The first step in converting a standard LP into the tableau (matrix) form is:

Example:

Consider the following "≤"-constraint:

This constraint implies that:

We must add an amount ≥ 0 to x₁ + x₂ to obtain the value 5

In other words:

x₁ + x₂ + (some value ≥ 0) = 5

Or:

x₁ + x₂ + s1 = 5 s1 ≥ 0

The variable s₁ is called a surplus variable

Remember:

Setting up the Simplex Tableau

Consider the following Linear constrained optimalization problem (or Linear Programming (LP) problem):

max: x₁ + x₂ s.t.: x₁ + 2x₂ ≤ 8 3x₁ + 2x₂ ≤ 12 x₁ ≥ 0, x₂ ≥ 0

We first write the objective function as an equation by introducing an objective value variable z:

max: z = x₁ + x₂ s.t.: x₁ + 2x₂ ≤ 8 3x₁ + 2x₂ ≤ 12 x₁ ≥ 0, x₂ ≥ 0

We want to maximize the variable x...

Setting up the Simplex Tableau:

Re-write the Linear Programming problem as a system of linear equations (=):

Original equation Converted equation --------------------+-------------------------------- z = x₁ + x₂ ⇒ z - x₁ - x₂ = 0 x₁ + 2x₂ ≤ 8 ⇒ x₁ + 2x₂ + s₁ = 8 (Add surplus variable) 3x₁ + 2x₂ ≤ 12 ⇒ 3x₁ + 2x₂ + s₂ = 12 (Add surplus variable)

Put the linear system into a matrix form:

Linear system: z - x₁ - x₂ = 0 x₁ + 2x₂ + s₁ = 8 3x₁ + 2x₂ + s₂ = 12
Lined up: (z is traditionally the last column) -x₁ - x₂ + z = 0 x₁ + 2x₂ + s₁ = 8 3x₁ + 2x₂ + s₂ = 12 Let's fill in all coefficients for completeness: -1x₁ - 1x₂ + 0s₁ + 0s₂ + 1z = 0 1x₁ + 2x₂ + 1s₁ + 0s₂ + 0z = 8 3x₁ + 2x₂ + 0s₁ + 1s₂ + 0z = 12
Re-written in Matrix form: x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12

Notes:

The order of the columns in a Simplex tableau is traditionally:

Original variables of the LP
They are named: x₁, x₂, ...
Artificially introduced variables needed to use the Simplex Algorithm
They are named: s₁, s₂, ... (surplus variables) and a₁, a₂, ... (artificial variables)
The objective variable z (which is the last variable in the list)
The Right Hand Size (RHS) consists of constant values

Example:

Original Surplus Obj variable var's func RHS <---------><-------><----> <----> x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12

The matrix form of the Linear Program is called:
Traditionally, the objective value z is always the last column of the Simplex Tableau

Property of a Simplex Tableau

Property that every Simplex Tableau must satisfy:

Explanation:

A column vector is a column of values

Example:

LHS of Simplex Tableau RHS -------------------------+------- -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 ^ ^ | | "column" vectors

A canonical unit column vector is a column vector where:

One of the value is equal to 1
All other values are equal to 0

Example:

LHS of Simplex Tableau RHS -------------------------+------- -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 ^ ^ ^ | | | "Canonical unit" column vectors

Observe that:

The Simplex tableau has 3 rows
There are also 3 canonical unit vectors in the Simplex tableau !!!

I I I I

Basic variables, basis and basic solutions

3 important concepts in the Simplex method:

Definition: Basic variable:

A basic variable is:

Related: Non-basic variable:

A non-basic variable is a variable that is not a basic variable

Example:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 Basic variables: s₁ s₂ z Non-basic variables: x₁ x₂

Definition: Basis

Definition: Basic solution

Basic solution = the solution found in ("associated with") a Simplex tableau by:

Example:

Consider the following Simplex Tableau:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 Basic variables: s₁ s₂ z Non-basic variables: x₁ x₂

This corresponds to this set of equations:

-1x₁ - 1x₂ + 0s₁ + 0s₂ + 1z = 0 1x₁ + 2x₂ + 1s₁ + 0s₂ + 0z = 8 3x₁ + 2x₂ + 0s₁ + 1s₂ + 0z = 12

To find the basic solution:

Set the non-basic variables (x₁ and x₂) to 0:

0 + 0 + 0s₁ + 0s₂ + 1z = 0 0 + 0 + 1s₁ + 0s₂ + 0z = 8 0 + 0 + 0s₁ + 1s₂ + 0z = 12 Or: 0s₁ + 0s₂ + 1z = 0 1s₁ + 0s₂ + 0z = 8 0s₁ + 1s₂ + 0z = 12

"Solve" it (there not much to solve really) ---result:

z = 0 s₁ = 8 s₂ = 12

This leads to a simple recipe on how to find the basic solution for a Simplex tableau:

Fact:

Every Simplex tableau is associated with a basic solution.
(That's because a Simplex Tableau with m rows always has m basic variables !!!)
See: the property of a Simpelx Tableau: click here

The locations of the basic solution in Simplex

Fact about the basic solutions in a Simplex tableau:

Example: the location of a basic solution

Simplex Tableau:

Matrix form: x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12

Basic solution:

x₁ = 0       (non-basic variable = 0))
x₂ = 0       (non-basic variable = 0))

s₁ = 8       (basic variable = RHS))
s₂ = 12     (basic variable = RHS))
z = 0        (basic variable = RHS))

Note:

Since the objective function z = x₁ + x₂, and

x₁ = 0       (non-basic variable = 0))
x₂ = 0       (non-basic variable = 0))

Naturally, the objective variable z = x₁ + x₂ is equal to 0 + 0 = 0 !!!

Graphical depiction of this basic solution: (x₁ = 0, x₂ = 0)

Review: invariant row operations in Linear Algebra

Consider the following system of equations:

2 x₁ + 4 x₂ = 16 3 x₁ + 2 x₂ = 12

The equality (=) means that the values in the left hand side and right hand side of "=" are in "balance".

The systems of equations above can be represented by this figure:

2 x₁ + 4 x₂ = 16 3 x₁ + 2 x₂ = 12

Invariant row operation 1:

You can multiply (or divide) both sides of an equation by the same factor, and the resulting equality will hold (= true).

Example: divide the first equation by 2 (= multiply first equation by 0.5)

0.5 × (2 x₁ + 4 x₂) = 0.5 × 16 ==> x₁ + 2 x₂ = 8

Invariant row operation2:

You can add (or subtract) a multiple of one equation to a second equation (the left hand side to left hand side and the right hand side to the right hand side). and the resulting equality will hold (= true)

Example: subtract 3 × the first equation from the second equation

x₁ + 2 x₂ = 8 (1) 3 x₁ + 2 x₂ = 12 (2) (2) ===> 3 x₁ + 2 x₂ = 12 3 x (1) ===> 3 x₁ + 6 x₂ = 24 - --------------------------- 0 x₁ - 4 x₂ = -12

Pivoting: going from a basic solution to a better basic solution

Pivoting:

Apply invariant row operations to the system of linear equations in the Simplex Tableau such that:

A non-basic variable will become a basic variable        and

A (some other) basic variable will become a non-basic variable

Example:    pivoting

The starting basic solution:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 Basic variables: s₁ s₂ z

Let's:

Make variable s₂ into a non- basic variable
make variable x₁ into a basic variable

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 Step 1: make the value under x₁ equal to 1 (Divide row by 3)
x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (Row 1) 1 2 1 0 0 | 8 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3) Step 2: make the other value under x₁ equal to 0 (Add a multiple of row 3 to the other rows)
After: row 1 = row 1 + 1 × (row 3) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 1 2 1 0 0 | 8 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3) After: row 2 = row 2 - 1 × (row 3) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 0 1.3 1 -0.3 0 | 4 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3)

The new Simplex Tableau:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 0 1.3 1 -0.3 0 | 4 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3)

The new set of basic variables:

x₁
s₁
z

The basic solution corresponding to the new Simplex Tableau is:

x₂ = 0      (non-basic variable)
s₂ = 0      (non-basic variable)

x₁ = 4      (basic variable, RHS)
s₁ = 4      (basic variable, RHS)
z = 4      (basic variable, RHS)

Graphical depiction of this basic solution: (x₁ = 4, x₂ = 0)

Note:

The basic solution is again a "corner" point in the feasible region

This basic solution is better than the previous one because:

The objective value z = 4
(The previous solution has an objective value z = 0)

Selecting a pivot that guarantee to improve on the objective function

Pivot:

The pivot is the element in a Simplex Tableau that will become the 1 (one) value of the new basic variable

Example:

Simplex Tableau before pivoting:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12

The high-lighted element (3) was the pivot is the row operations
Simplex Tableau after pivoting:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 0 1.3 1 -0.3 0 | 4 1 0.7 0 0.3 0 | 4

Mathematics behind the selection of the pivot:

I will omit the Mathematical theory behind the how to select the pivot
(Requires matrix manipulations)

I will only show you rules on how to pick the pivot
The pivot selection rules are based on 2 Mathematical results:

The pivot column is selected by an optimality criterium

The pivot row is selected by a feasibility criterium

Pivot picking rules:

Find the largest negative cofficient associated with a non-basic variable in the first (objective value) equation
Reason:

Choosing the largest negative value will make the objective function increase the fastest

(Break ties aribitrarily)
Example:

Simplex Tableau:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (row 1 = objective function) 1 2 1 0 0 | 8 3 2 0 1 0 | 12

Non-basic variables:

x₁      coefficient = −1
x₂      coefficient = −1

Selection:     x₁      (tie broken arbitrarily)

Compute all the test ratio "RHS_i/coef_i" for each coef_i > 0 under the chosen non-basic variable

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (neg. coef - skip it) 1 2 1 0 0 | 8 test ratio = 8/1 = 8 3 2 0 1 0 | 12 test ration = 12/3 = 4

The pivot element is the coefficient that has the smalles test ratio:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (neg. coef - skip it) 1 2 1 0 0 | 8 test ratio = 8/1 = 8 3 2 0 1 0 | 12 test ration = 12/3 = 4

Pivot operation:

After determining the pivot element, a pivot operation is performed:

Divide the pivot row by the value of the pivot element

Sweep the pivot column clear (make the other elements in the pivot column = 0 (zero))

The Simplex Algorithm

Simplex Algorithm:

Find an initial Simplex Tableau that contains a basic solution

The initial basic solution will produce a objective value = 0 (zero)

Repeat the steps:

Find a pivot

Perform the pivot operation

until all coefficients in the objective function row are ≥ 0
(I.e., no more pivot candidates)

This step will improve the objective function

When there are no more pivot candidates, the objective function is optimized

A complete example

Linear Program:

max: z = x₁ + x₂ s.t.: x₁ + 2x₂ ≤ 8 3x₁ + 2x₂ ≤ 12 x₁ ≥ 0, x₂ ≥ 0

Step 1: transform to system of linear equation with a basis

Add surplus variables: z - x₁ - x₂ = 0 x₁ + 2x₂ + s₁ = 8 3x₁ + 2x₂ + s₂ = 12
Lined up: -x₁ - x₂ + z = 0 x₁ + 2x₂ + s₁ = 8 3x₁ + 2x₂ + s₂ = 12
In Tableau form: x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 Basis: s₁ s₂ z

This is the initial Simplex Tableau

Pivot step 1:

Select the pivot column using the largest negative coefficient in the first (objective) row:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 largest negative value 1 2 1 0 0 | 8 3 2 0 1 0 | 12

Select the pivot row using the smallest test ratio associated to a positive coefficient:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (neg. coef - skip it) 1 2 1 0 0 | 8 test ratio = 8/1 = 8 3 2 0 1 0 | 12 test ration = 12/3 = 4

Pivot:

Step 1: pivot row by pivot element x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 3 2 0 1 0 | 12 | V x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 1 2 1 0 0 | 8 1 0.7 0 0.3 0 | 4
Step 2: sweep the pivot column x₁ x₂ s₁ s₂ z | RHS ==========================+==== -1 -1 0 0 1 | 0 (Row 1) 1 2 1 0 0 | 8 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3) After: row 1 = row 1 + 1 × (row 3) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 1 2 1 0 0 | 8 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3) After: row 2 = row 2 - 1 × (row 3) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 0 1.3 1 -0.3 0 | 4 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3)

Corresponding basic solution:

x₁ = 4
x₂ = 0

Objective value: z = 4

Graphically:

Pivot step 2:

Select the pivot column using the largest negative coefficient in the first (objective) row:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (Row 1) 0 1.3 1 -0.3 0 | 4 (Row 2) 1 0.7 0 0.3 0 | 4 (Row 3)

Select the pivot row using the smallest test ratio associated to a positive coefficient:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 0 1.3 1 -0.3 0 | 4 test ratio = 4/1.3 = 3 1 0.7 0 0.3 0 | 4 test ratio = 4/0.7 = 6

Pivot:

Step 1: pivot row by pivot element x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 0 1.3 1 -0.3 0 | 4 divide row by 1.333333 1 0.7 0 0.3 0 | 4 | V x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 0 1 0.75 -0.25 0 | 3 1 0.7 0 0.3 0 | 4
Step 2: sweep the pivot column x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 -0.3 0 0.3 1 | 4 (row 1) 0 1 0.75 -0.25 0 | 3 (row 2) 1 0.7 0 0.3 0 | 4 (row 3) After: row 1 = row 1 + 0.333333 × (row 2) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 0 0.25 0.25 1 | 5 (row 1) 0 1 0.75 -0.25 0 | 3 (row 2) 1 0.7 0 0.3 0 | 4 (row 3) After: row 3 = row 3 - 0.6666666 (0.7) × (row 2) x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 0 0.25 0.25 1 | 5 (row 1) 0 1 0.75 -0.25 0 | 3 (row 2) 1 0 -0.5 0.5 0 | 2 (row 3)

Corresponding basic solution:

x₁ = 2
x₂ = 3

Objective value: z = 5

Graphically:

Pivot step 3:

Select the pivot column using the largest negative coefficient in the first (objective) row:

x₁ x₂ s₁ s₂ z | RHS ==========================+==== 0 0 0.25 0.25 1 | 5 (row 1) - no negative coefficient 0 1 0.75 -0.25 0 | 3 (row 2) 1 0 -0.5 0.5 0 | 2 (row 3)

DONE !!!
(Optimal solution found)