What is machine learning regression

Difference between linear regression in machine learning and the statistical model


I had the understanding that the main difference between machine learning and the statistical model is that the later "assumes" a certain kind of data distribution and is based on this different model paradigm as well as statistical results that we get (e.g. p-values, F-statistic), t-stat, etc.). In machine learning, however, we don't care about the distribution of data and we are more interested in predictions.

While looking through mllib doc, I found that we are giving a distribution for linear regression. But Mllib is a machine learning package. So I have the following questions:

1) Is my understanding between ML and statistical method wrong?

2) Does Spark use statistical models for linear regression and GLMs?

Many Thanks!

Note: There are many wonderful posts on the difference between machine learning and statistical methods. However, this is more related to the MLLIB spark.

Reply:


  1. Unfortunately, the dichotomy you have described is invalid. ML models (almost always) define a distribution of responses. For example, the extremely popular XGBoost machine library defines certain learning objectives (e.g. linear, logistic, Poisson, Cox, etc.) to increase the gradient.
  2. The implementation of linear regression and GLMs in Spark's MLlib is based definitely on standard statistical theory for linear models. For example, quote directly from the comments: This is a standard linear regression algorithm for the Gaussian answer. The implementation of each algorithm could be tweaked to work for very large amounts of data (see for example this excellent thread on "Why use gradient descent for linear regression when a closed-form math solution is available?"), But the one behind it standing theory An algorithm is exactly the same.



We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By continuing, you consent to our use of cookies and other tracking technologies and affirm you're at least 16 years old or have consent from a parent or guardian.

You can read details in our Cookie policy and Privacy policy.