Understanding The Efficiency Of GPU Algorithms For Matrix-Matrix Multiplication And Its Properties.

Understanding The Efficiency Of GPU Algorithms For Matrix-Matrix Multiplication And Its Properties.

Definition of a Matrix 

 

An array of numbers/ variables is called a matrix. The horizontal arrays of a matrix are called its rows and the vertical arrays are called its columns. Let M is a matrix having m no.of rows and n no.of columns. Then, A is said to have an order of m×n. Where m is no. of rows and n is no columns and can be represented in either of the following forms.

Matrix multiplication

Let’s understand what a matrix really is..

 

a matrix over a field X is a rectangular array of scalars, each of which is a member of X. basically in mathematics matrix is nothing but a rectangular array or we can say a table of numbers, expressions, symbols etc. arranged in rows and columns the size of a matrix is the number of rows and columns it contains here in example A is 22 matrix. 

 

Now there are many terminologies like, if the matrix contains only a single row then it is called a row matrix.

 

When the matrix contains a single column it is called a column matrix sometimes also called vectors.

 

 

A matrix with the same number of rows and columns is called a square matrix.

 

There are many more, but remember a 2D matrix in coordinate system depicts a parallelogram in coordinate system.

A 3D matrix in a coordinate system depicts a parallelepiped i.e combination of 6 parallelograms . 

 

Now, these were some awesome facts about a matrix. Lets jump into the multiplication of a matrix.

2Matrix multiplication

What do you understand from the matrix multiplication process and why GPU?

  • Remember matrix is actually a 2 dimensional array, but it’s the key point for shifting to GPU instead of CPU for deep learning, let this issue be detailed.

  • Lets focus on multiplication of two matrices and its process.

3 Matrix multiplication

See, in the above picture , to get the resulting matrix which is C ( b X d ) in this case, we have to multiply with i no.of  rows to j no. of columns + we have to add the previous one result.

 

The above process is very time consuming, that’s why any data scientist or machine learning programmer/ enthusiast who has been trying to train models at scale will at some point hit a enter button, and start to experience various degrees of processing lag. And when the data set is bigger, they may have to wait till weeks. So, they shift to GPU instead of CPU.

 

Many will tell you Deep learning requires a lot of computational power to run on. But it’s false, it doesn’t need computational power. Instead it can run easily on my Windows 7 old fashion computer also, but it will require more time as it has to multiply big matrices.

For any neural network, the training phase of the deep learning model is the most resource-intensive task, while training neural networks takes input and then they are processed in hidden layers etc. but these are nothing but matrix multiplication that you read a couple of minutes ago.

In a neural network, the first array is the input to the neural network, and the second array forms its weight

So, far so good.. Lets focus on some matrix multiplication properties.

 

Properties of Matrix Multiplication

For any  a x b matrix A there is an a x b matrix B (valued -A) with

A + B  =  O (null matrix of order a x b)    additive inverse

Let A, B and C be matrices of any dimensions then, these properties are defined–

 A(BC)  =  (AB)C                 associative

             A(B + C)  =  AB + AC        distributive

             (A + B)C  =  AC + BC        distributive 

There are unique matrices C and D with,  C A  =  A D  =  A   and it is called multiplicative identity.

One, general thing to note matrix multiplication doesn’t support commutative property i.e, AB BA. maybe it is sometimes equal but it doesn’t always mean it will be.

Now, what’s about the computational complexity of multiplication? It’s a huge field of interest for sake of knowledge i am just telling you the complexity its O( n3 ).

Article By: Satyam Roy

If you are Interested In Machine Learning You Can Check Machine Learning Internship Program

Also Check Other technical and Non Technical Internship Programs

Facebook
WhatsApp
LinkedIn
Email