Tensors: Data Structure for Deep Learning
This is a short post on what are tensors and its 3 important attributes.
Whether it’s a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Reinforcement etc, we have to provide some input to the algorithm. Most probably these will be numerical inputs. For instance, an input for an image classification might be number of pixels and the RGB value.
In a traditional programming language, we usually initialize input variables of specific types like String, Boolean, Integer, Float, Arrays, Map, custom data types etc and assign values to these variables which will be used in the later portion of the program. In case of deep learning (or machine learning) we will be working with complex and large datasets and it becomes important to present this input data in a proper structure that can be fed to the neural network and at the same time which is also representable.
In mathematics, a Tensor is a data structure used in Linear Algebra and is represented as an n-dimensional array.
Let’s refresh few terms of Linear Algebra and Geometry.
Dimension
It is the measurement of the size of an object or the distance of an object in one direction.
Plane
In mathematics, a plane is a flat, two-dimensional surface that extends infinitely far. The 2 dimensions are denoted by x and y and are called as parameters.
Scalars (0-dimensional tensors)
Any quantity that can be represented by a Real Number is called a Scalar. That means the height of a person, weight of a brick, temperature of the room etc are all scalar quantities. This number can be positive or negative. For instance, a temperature reading of -18C is a valid scalar quantity. You can also say that a scalar quantity is just a Point since it has got only magnitude/size and no length, breadth or height and hence it has 0 dimensions.
In deep learning, a tensor that contains only 1 number is called a scalar. If you notice carefully we don’t call it a number but we say that it’s a tensor of 0 dimension that contains a number of integer or float types. This means that even though we are dealing with Real numbers here, we usually won’t use it as raw input (integer or float type) but rather convert it into the tensor type and use it. Libraries such as NumPy help us to do that.
The values of x0, x1 and x2 are integers. The only difference is in the data type. x0 is of an int type whereas x1 and x2 are of type numpy.int32 and numpy.ndarray respectively. Since numpy converted x1 and x2 to tensor we are able to execute functions like ndim (numpy function to get the dimension of a tensor) and shape (I will explain this below). This cannot be done for x0 since it is not a tensor and doesn’t contain dimension and shape properties.
In this case, since x1 and x2 are scalar we got dimension as 0 and shape as none but the point is these variables are tensors.
Vectors (1-dimensional tensors)
A quantity that is described by its magnitude/size and in addition to that needs another parameter called direction, to describe itself, is called a vector. A simple example would be Force. Force has got a magnitude (the amount of force applied on an object) and a direction in which it is applied. If we could plot this quantity on a horizontal plane, we will start with a point (scalar with a magnitude) and stretch until we have reached another point, we would get a line segment that then describes this quantity. The amount of stretch that we had to draw gives us the length/direction and the horizontal plane is called its axis. Since we need only one axis to describe it, it’s one dimensional.
Back to deep learning: An array of numbers is called a vector. In other words, to describe a vector you need to use an array of scalars. Let’s take can example of color. In computer graphics color is a vector quantity. For instance, a RGB color system consists of 3 scalar quantity Red, Green and Blue. The combined intensity of these 3 components determines the output color.
Matrices (2 dimensional tensors)
A Matrix consists of an array of vectors and has got 2 axes which are usually referred to as rows and columns.
The most simplest example (and commonly used one) is that of a display unit, example: computer monitor, LCD screen, television set etc. The screen is a matrix of fixed number of rows and columns(1920x1080, 1024x768 etc) called as its resolution . If rows and columns could be denoted as x and y respectively, then at each intersection of (x, y) on the screen, we get a Picture Element or Pixel: the basic unit of any image displayed on the screen. For a grayscale image the value of a pixel (or its intensity) will range any where between 0–255.
If we consider a very tiny image of 10x10 pixels, it can be represented as:
3D, 4D and 5D ( higher dimensional tensors)
If we have got an array of 2D tensors, it will form a 3D tensor. Similarly if we have an array of 3D tensor it will form a 4D tensor and an array of 4D tensor will form a 5D tensor. Higher dimension tensors are used for processing highly complex tasks such as Image and Video processing.
Rank, Shape and Data type
A tensor has 3 attributes:
- Rank or Dimension: The number of axis of a tensor is called as its Rank. For a scalar it is 0, vector it is 1, for matrix it is 2 and so on. The ndim function of numpy library that we used above shows the rank or dimension of a given tensor.
- Shape: For a given tensor of a specific rank, its shape will be the number of elements ( or also called as dimension but this is different from rank) along its each axis. For scalar example we saw that shape is 0 because it doesn’t have any axis. For the vector example, the shape is shown as 3 because, it has 1 axis with 3 elements in it. For the matrix example the shape is (10,10) because it has 2 axis and each axis contains 10 elements. For the 3d tensor example the shape is (4,2,4) because it has 3 axis and contains 4 elements in first, 2 elements in second and 4 elements in third axis.
- Data type: This is the type of the data contained in the tensor.
Closing notes
I have just started studying Deep Learning and it took sometime for me to understand the concept of Tensors. The post was an effort to document what I have learnt so far about tensors and to share it with the community.
In my next article, I will share few examples of mathematical operations that can be performed on tensors.
Comments and inputs are welcome. Thanks for taking time to read this post!
References
- Deep Learning with Python: Francois Chollet
- Linear Algebra: Wikipedia
