Perceptron

In this notebook, we will construct simple perceptron models. We’ll start by implementing a perceptron model, and seeing how it behaves. We’ll then outline the steps to train a perceptron to classify a point as above or below a line.

This discussion follows the excellent example and discussion at The Nature of Code. Please see that reference for additional details, and a more sophisticated coding strategy (using Classes in Python).

1 Preliminaries

Before beginning, let’s load in the Python packages we’ll need:

2 A simple perceptron model

Let’s examine a simple perceptron that accepts inputs, processes those inputs, and returns an output. To do so, please complete this function:

Q: How many inputs does the function take? How many outputs does it return?

Q: Apply this function to different sets of inputs. Consider,

input1 = 1, input2 = 0, w1 = 0.5, w2 = -0.5, theta = 0

and

input1 = 1, input2 = 0, w1 = 0.5, w2 = -0.5, theta = -1

What do you find?

3 Build a perceptron classifier

We’d like to create a method to train a perceptron to classify a point (x,y) as above or below a line. Let’s implement this training procedure.

3.1 Provide perceptron with inputs and known answer.

First, let’s make a function that computes a line, and determines if a given y value is above or below the line. We’ll use this function to return the correct (“known”) answer. Having known answers is important for training the perceptron. We’ll use the known answers to tell the when it’s right or wrong (i.e., when the perceptron makes an error).

Let’s define the function (known_answer) should take four inputs:

  • slope
  • intercept
  • x
  • y

where the (x,y) value is a point we choose on the plane. The function should return one output:

  • desired_output

where,

desired_output = 1, if the y value (the last input) is above the line,

desired_putput = 0, if the y value (the last input) is below the line.

Complete the function below:

Q: Consider the (x,y) point,

x,y = 0.7,3

and the line with slope and intercept,

slope = 2

intercept = 1

Is the (x,y) point above or below the line?

A: To answer this, let’s ask our function,

A (Continued): We find a correct_answer of 1.

So, the point (x,y)=(0.7,3) is above the line with slope 2 and intercept 1.

3.2 Ask perceptron to guess an answer.

Our next step is to compare our desired output (computed in Step 1) to the output guessed by the perceptron. To do so, we’ll need to compute the feedforward solution for the perceptron (i.e., given the inputs and bias, determine the perceptron output). Let’s do so complete the function below,

This function takes five inputs:

  • x = the x coordinate of the point we choose in the plane.
  • y = the y coordinate of the point we choose in the plane.
  • wx = the weight of x input.
  • wy = the weight of y input.
  • wb = the weight of the bias.

And this function returns one output:

  • the perceptron’s guess, is the point above (=1) or below (=0) the line.
Q: Again consider the (x,y) point,

x,y = 0.7,3

and set initial values for the perceptron weights. Let’s just set these all to 0.5; our goal in the rest of this module will be to train the perceptron by adjusting these weights. But for now,

wx,wy,wb = 0.5

Then, ask the perceptron for it’s guess for it’s guess, is the point above or below the line?

A: We find a peceptron_guess of 1.

So, the perceptron guesses that the point (x,y)=(0.7,3) is above the line.

3.3 Compute the error.

We’ve now answered the question “Is the (x,y) point above the line?” in two ways:

  • the known answer, and
  • the perceptron’s guess.

Let’s compute the error as the difference between these two answers:

Q: What do you find for the error? Does it make sense?

3.4 Adjust all weights according to the error.

To update the weights, we’ll use the expression,

new weight = weight + error * input * learning constant

We need to compute this for each weight (wx, wy, wb).

First, let’s set the learning constant,

Then, we can compute the new weights,

Notice that, in the update to wb we use the fact that the bias equals 1.

Q: What do you find for the new weights? Does it make sense?

3.5 Return to Step 1 and repeat …

We could try to compute these repetitions by hand, for example by repeating the cells above. To do so, we’d choose a new point in the (x,y) plane, determine whether it’s above the line 2x+1, ask the perceptron to guess whether it’s above the line, then use the error to update the perceptron’s weights.

But we want to evaluate this procedure 2000 times. Doing so by hand would be a total pain, and highly error prone. Instead, let’s ask the computer to do the boring work of multiple repetitions. To do so, let’s collect the code above, and examine 2000 (x,y) points chosen randomly in the plane. We’ll wrap our code above inside a for-loop to make this efficient,

4 Challenges

1. Load the file training_data.csv available on the course repository. You will find that this file contains the following variables:

Training Data:

x_training = x-coordinate to train the network

y_training = y-coordinate to train the network

correct_answer = vector of 0’s and 1’s.

These three variables define the training data. For each (x,y)-coordinate, the correct_answer indicates whether the point is above (1) or below (0) the line. There are 2000 example (x,y)-cooridates, each with a correct classification.

Load the file testing_data.csv available on the course repository. You will find that this file contains the following variables:

Testing Data:

x_testing = x-coordinate to test the network

y_testing = y-coordinate to test the network

The challenge is to determine whether each (x,y)-coordinate in the testing data is above or below the line. To do so, train a perceptron using the training data. Then, apply your trained perceptron to the testing data.

  1. How many points in the testing data are above the line, below the line?

  2. What is the slope of the line?

  3. What is the intercept?

2. Load the file training_data_3.csv available on the course repository. You will find that this file contains the following variables:

Training Data:

x_training = x-coordinate to train the network

y_training = y-coordinate to train the network

z_training = z-coordinate to train the network

correct_answer = vector of 0’s and 1’s.

There are 1000 example points each with a correct classification, which indicates whether the point is above (1) or below (0) a plane.

Load the file testing_data_3.csv available on the course repository. You will find that this file contains the following variables:

Testing Data:

x_testing = x-coordinate to test the network

y_testing = y-coordinate to test the network

z_testing = z-coordinate to test the network

The classification for these points, as either above or below the plane, is unknown.

The challenge is to correctly classify each point in the testing data as above or below the plane. To do so, train a perceptron using the training data. Then, apply your trained perceptron to the testing data.

  1. How many points in the testing data are above the plane, below the plane?

  2. Can you use the results of your trained perceptron to write an equation for the plane? Hint: write the equation in this form,

    a x + b y + c z = d

and determine the unknown parameters a, b, c, d.