Computer Vision and OpenCV

This notebook is a part of AI for Beginners Curriculum.

OpenCV is considered to be de facto standard for image processing. It contains a lot of useful algorithms, implemented in C++. You can call OpenCV from Python as well.

In this notebooks, we will give you some examples for using OpenCV. For more details, you can visit Learn OpenCV online course.

First, let's import cv2, as well as some other useful libraries:

Loading Images

Images in Python can be conveniently represented by NumPy arrays. For example, grayscale image with size of 320x200 pixels would be stored in 200x320 array, and color image of the same dimension would have shape of 200x320x3 (for 3 color channels).

Let's start by loading an image:

As you can see, it is an image of braille text. Since we are not very interested in the actual color, we can convert it to black-and-white:

Braille Image Processing

If we want to apply image classification to recognize the text, we need to cut out individual symbols to make them similar to MNIST images that we have seen before. This can be done using object detection technique which we will discuss later, but also we can try to use pure computer vision for that. A good description of how computer vision can be used for character separation can be found in this blog post - we will only focus on some computer vision techniques here.

First, let's try to enhance the image a little bit. We can use the idea of thresholding (well described in this OpenCV article):

To work with images, we need to "extract" individual dots, i.e. convert the images to a set of coordinates of individual dots. We can do that using feature extraction techniques, such as SIFT, SURF or ORB:

Let's plot all points to make sure we got things right:

To separate individual characters, we need to know the bounding box of the whole text. To find it out, we can just compute min and max coordinates:

Also, this text can be partially rotated, and to make it perfectly aligned we need to do so-called perspective transformation. We will take a rectangle defined by points $(x_{min},y_{min}), (x_{min},y_{max}), (x_{max},y_{min}), (x_{max},y_{max})$ and align it with new image with proportional dimensions:

After we get this well-aligned image, it should be relatively easy to slice it into pieces:

You have seen that quite a lot of tasks can be done using pure image processing, without any artificial intelligence. If we can use computer vision techniques to make the work of a neural network simpler - we should definitely do it, because it will allow us to solve problems with smaller number of training data.

Motion Detection using Frame Difference

Detecting motion on video stream is a very frequent task. For example, it allows us to get alerts when something happens on a surveillance camera. If we want to understand what's happening on the camera, we can then use a neural network - but it is much cheaper to use neural network when we know that something is going on.

The main idea of motion detection is simple. If the camera is fixed, then frames from the camera should be pretty similar to each other. Since frames are represented as arrays, just by subtracting those arrays for two subsequent frames we will get the pixel difference, which should be low for static frames, and become higher once there is substantial motion in the image.

We will start by learning how to open a video and convert it into a sequence of frames:

Since color is not so important for motion detection, we will convert all frames to grayscale. Then, we will compute the frame differences, and plot their norms to visually see the amount of activity going on:

Suppose we want to create a report that shows what happened in front of the camera by showing the suitable image each time something happens. To do it, we probably want to find out the start and end frame of an "event", and display the middle frame. To remove some noise, we will also smooth out the curve above with moving average function:

Now we can find out frames that have the amount of changes above the threshold by using np.where, and extract a sequence of consecutive frames that is longer than 30 frames:

And finally we can display the image:

You may notice that color scheme on this image does not look right! This is because OpenCV for historical reasons loads images in BGR color space, while matplotlib uses more traditional RGB color order. Most of the time, it makes sense to convert images to RGB immediately after loading them.

Extract Motion using Optical Flow

While just comparing two consecutive frames allows us to see the amount of changes, it does not give any information on what is actually moving and where. To get that information, there is a technique called optical flow:

• Dense Optical Flow computes the vector field that shows for each pixel where is it moving
• Sparse Optical Flow is based on taking some distinctive features in the image (eg. edges), and building their trajectory from frame to frame.

Read more on optical flow in this great tutorial.

Let's compute dense optical flow between our frames:

As you can see, for each frame the flow has the dimension of the frame, and 2 channels, corresponding to x and y components of optical flow vector.

Displaying optical flow in 2D is a bit challenging, but we can use one clever idea. If we convert optical flow to polar coordinates, then we will get two components for each pixel: direction and intensity. We can represent intensity by the pixel intensity, and direction by different colors. We will create an image in HSV (Hue-Saturation-Value) color space, where hue will be defined by direction, value - by intensity, and saturation will be 255.

So, in those frames greenish color corresponds to moving to the left, while blue - moving to the right.

Optical flow can be a great tool to draw conclusions about general direction of movement. For example, if you see that all pixels in a frame are moving in more or less one direction - you can conclude that there is camera movement, and try to compensate for that.