Algorithm Analysis¶

Preliminaries¶

An Algorithm is simply a set of instructions. Genreally, the instructions must be specific enough that one can estimate the amount of work required to complete each step. Multiple algorithms can exist to solve the same problem or complete the same task. How do we choose which algorithm to use?

The appropriate algorithm can be determined based on an number of factors:

How long the algorithm takes to run
What resources are required to execute the algorithm
How much space or memory is required
How exact is the solution provided by the algorithm (is estimation involved, what about floating point rounding?)

Lets look at an example, the 3 sum problem. The 3 sum problem is simple, given an array of numbers, how many triples of those numbers sum to the value 0:

int count(long[] a)
{
    int N = a.length;
    int cnt = 0;
    for (int i = 0; i < N; i++)
        for (int j = i+1; j < N; j++)
            for (int k = j+1; k < N; k++)
                if (a[i] + a[j] + a[k] == 0)
                    cnt++;
    return cnt;
}

3-Sum analysis

How do we expect the program to run?
How do we quantify this expectation?
What the properties of this algorithm do the quantaization of expected running time apply to?

Running time is one way to quantify our expectation:

Best case: the easiest inputs. The best we can do
Worst case: The hardest inputs. The worst we can do
Average case: What we expect on average, for random input

Note that best case doesn’t mean just a small input. Rather we can consider differeing inputs of the same size with varying hardness

What is the expected best case, worst case, and average case running time?

Here is 3-sum running on home.cs. Why is 300 doing the wrong thing?:

Just to get an idea of running time and order:

Those numbers are nice, but they are hiding something. What are they hiding?

2 Different 3-Sum implementations

In the following are 2 different 3-sum implementations.

int count(long[] a)
{
    int N = a.length;
    int cnt = 0;
    for (int i = 0; i < N; i++)
        for (int j = 0; j < N; j++)
            for (int k = 0; k < N; k++)
                if (a[i] + a[j] + a[k] == 0)
                    cnt++;
    return cnt;
}

int count(long[] a)
{
    int N = a.length;
    int cnt = 0;
    for (int i = 0; i < N; i++)
        for (int j = i+1; j < N; j++)
            for (int k = j+1; k < N; k++)
                if (a[i] + a[j] + a[k] == 0)
                    cnt++;
    return cnt;
}

rhs = lower code

lhs = upper code

These were run on the same computer. Why is there such a large difference? what is the order of each algorithm?

Asymptotic Notation (Big-O and the others)¶

Recall, we were interested in describing the best, average, and worst case performacne of an algorithm. For now, lets concentrate on just the worst case performance. To describe the worst case perfoamnce of an algorithm, we need to represent its work in terms of a function that takes a parameter defining the input size. For example, the 3-sum problem described by a cubic function whose parameter is the length of the input array. 3sum = $3sum( n ) = n^3$ .

To describe the worst case behavior of an algorithm , we need to identify a function that grows at the same rate as the expected running time of the algorithm. In the worst case scenario, we us O notation.

Definition: Big-O

A function $g(n)$ is said to be $O(f(n))$ if there are positive constants $c$ and $n_0$ such that $g(n) \leq c f(n)$ for any $n \geq n_0$ .

In other words, $g(n)$ is bounded above by $f(n)$ .

Some questions...

Big O is the worst case performance of an algorithm. We can also model best case, and average case. Mathematically, the varios cases can be defined as follows:

Big O is a tight upper bound, while little O is a loose upper bound. In other words, if $g(n) = O(f(n) )$ , then there is no function $t(n)$ that bounds $g(n)$ from above but has a smaller order than $f(n)$ .

For example, the function $g(n) = n+3$ is $O(f(n) = n)$ and $o( t(n) = n^2+4$ .

Here are some examples to illustrate:

Examples of asymptotic analysis

$\begin{array}{l l}a(x) &= x^2 + x \\b(x) &= x^2/2 \\a(x)/b(x) & = \frac{x^2+x}{.5x^2} \\& = \frac{x^2}{.5x^2} + \frac{x}{.5x^2} \\& = \frac{1}{.5} + \frac{1}{.5x} \\& = 2 + \frac{2}{x}\\ & = 2(1+\frac{1}{x}) \end{array}$

plug in some big numbers for x, what happens?

NOTE: $a(x)/b(x) = 2$ as the limit apporaches infinity. So, it fits the definition of a tight upper bound, tight lower bound, and average case bound. This is the case with the 3Sum analysis as well.

Another example:

$\begin{array}{ll}a(x) &= x^2 + x \\c(x) &= x\\a(x)/c(x) &= \frac{x^2+x}{x}\\&= \frac{x^2}{x}+\frac{x}{x}\\&= x+1\end{array}$

We do not have a limit = $\infty$ , but we do have a limit = 0. Limit = $\infty$ is essentially the opposite of little o. We could call this little omega, $a(x)$ is loosly bounded below by $c(x)$

What do the loose bounds actually tell us?

Excersise

Identify an algorithm that has different best case and worst case bounds for the same size input.

The Usual Suspects

The following functions often come up in asymptotic analysis

$f(n)=k$ (constant)

$f(n) = k \lg n$ (logarithmic)

$f(n) = kn$ (linear)

$f(n) = kn \lg n$ (linearithmic)

$f(n) = kn^2$ (quadratic)

$f(n) = k2^n$ (exponential)

Finding the functions for an algorithm¶

This discussion is quite nice, but we must come up with a model of converting algorithms to functions. To do this, we must make an assumption:

Every basic operation takes 1 time unit to complete on a computer.

Lets define a basic operation as an arithmetic operation, a comparison operation, an assignment operation, a return from a function, etc. In general, if it can be represented by a low level assembly language instruction, we assume its implemented in 1 time unit.

1 time unit per operation?

Is the above assumption actually true? Provide some examples of when it is and when it isn’t. (hint, memory operations? floating point operations?, pipelining?)

given the above assumption, we can simply add up the number of operations a code segment takes to execute. Typically, it will vary based on input size.

Loop

for (i = 0; i < n; i++)      // 1+(n+1)+ n
    Z = 8 * i + 5;           // 3*n

Running time is $2n + 2 + 3n = O(n)$

Nested Loop

for (i = 0; i < n; i++)        // 1+(n+1)+n
    for (j = 0; j < n; j++)    // n * (1+(n+1)+n)
        x += 4 * (i + j)       // n * n * (4)

Running Time is:

$2 + 2n + 2n^2 + 2n + 4n^2 = 6n^2+4n+2 = O(n^2)$

Consecutive Statments

Just add all the consecutive statements together. (Or just keep the biggest part)

for (i = 0; i < n; i++)      // 1+(n+1)+ n
    Z = 8 * i + 5;           // 3*n
for (i = 0; i < n; i++)        // 1+(n+1)+n
    for (j = 0; j < n; j++)    // n * (1+(n+1)+n)
        x += 4 * (i + j)       // n * n * (4)

$6n^2+4n+2+ 2n + 2 + 3n = 6n^2+ 9n +4= O(n^2)$

Conditionals: (if statments)

Use the most expensive branch:

if( x < 2 )                        // 1
    x = 2;                         // 1
else {
    for (i = 0; i < n; i++)        // 1+(n+1)+n
        for (j = 0; j < n; j++)    // n * (1+(n+1)+n)
            x += 4 * (i + j)       // n * n * (4)
}

Running Time is: $2 \mathit{~OR~} 6n^2+4n+2 = O(n^2)$

Recursion

Recursion is trickier. Basically, you have to find the worst possible cost of a recrusive case, and the worst possible cost of a non-recursive case (base case). Then you have to find out how many times a recursive case is called in a given function call. An example follow in the next section.

Recurrence Relations For Recursion¶

A recurrence relation is simply a function that describes the cost of a recursive function call in an algorithm. Because the function calls itself, we can plug the function into itself, and generate a series. The time complexity of the function is equivalent to the convergence of the series. An example follows

Simple Example

int powerOf2(const int &n) {
    if (n == 0)                            // 1
        return 1;                          // 1
    return powerOf2(n-1) + powerOf2(n-1);  // 4 (+1 to fail at the IF)
}

The base case through function, when $n = 0$ , takes 2 time units.

The recursive case takes 5 time units, 1 to fail at the IF statement, and 4 to complete the function calls and add their results.

So, we can define a function $t(n)$ that behaves as follows:

$t(0) = 2$

$t(k) = 5 + 2(T(k-1))$ for $k>0$ (there are 2 function calls)

So, we can start plugging in values for the function:

$t(n) = 5 + 2T(n-1) = 5 + 2(5 + 2t(n-2)) = 5 + 2(5 + 2(5 + 2(t(n-3))))$

$= 5(1 + 2 + 2^2 + 2^3 + … + 2^{n-1}) + 2^nt(0) = O(2^n)$

Another Example

int betterPowerOf2(const int &n) {
    if (n == 0)
        return 1;
    return 2*betterPowerOf2(n-1);
}

$t(0) = 2$

$t(k) = 4 + t(k-1)$ for $k > 0$

$t(n) = 4 + t(n-1) = 4 + (4 + t(n-2)) = 4+ (4 + (4 + (t(n-3)))) = …$

$= 4n + t(0) = 4n + 2= O(n)$

Table Of Contents

Previous topic

Next topic

This Page

Algorithm Analysis¶

Preliminaries¶

Asymptotic Notation (Big-O and the others)¶

Finding the functions for an algorithm¶

Recurrence Relations For Recursion¶