I think of machine learning as the process of finding the underlying function which mapped one set of data to another in order to apply that function to more data from the original set.
(I wrote another post about this here.)
As such, you always have to make some useful assumptions about your model (assumptions which are almost always false to some degree, but nonetheless necessary for making a useful model.) One of the bigger assumptions you can make is that your model is "linear." A function is "linear" if it can be written as follows:
f(x) = a1*x1 + a2*x2 + a3*x3 + ... + an*xn
(where each "a" term is a constant and each "x" term is a "feature variable.")
[Equivalently "linear" actually means that f(a+b) = f(a)+f(b)]
As an example let's consider a simple dataset...let's say there's some dataset of people and we want to predict weight using height and age. If we assumed a linear model, then we would have the following:
weight(x) = a1*height(x) + a2*age(x)
and the goal would be to find the best a1, and a2 to match the data we'd been given.
For linear models, it is actually remarkably simple to find THE best solution:
We make a matrix, M, where each row is [height,age] for a new person in our dataset.
Then we have X = [a1,a2] be a column vector and B be the vector of all weights for each person in our dataset.
This gives the following matrix equation:
MX = B
Now if M is a square matrix, then you can multiply on the left by M^(-1) to solve for X, but chances are that we have way more people in our dataset than we have feature variables. (The matrix probably won't be square.) In this case, we can use Gaussian Elimination to try and solve it...but it probably won't have a solution...so we can instead solve the following by Gaussian Elimination:
MX = C
Where C is the projection of B onto the "column space" of M. (This means that it's the closest thing to B that makes MX=C have a solution.)
Now this has all been pretty basic linear algebra so far, but it always helps to review. (It's been a while for me too so let me know if I made any glaring mistakes)
I just found this cool 3blue1brown video series on linear algebra which could help to review ;)