February 12, 2020

Prof. Steinwart, what does machine learning have to do with mathematics?

Interview with members of the Department of Mathematics

In machine learning one considers computer algorithms that learn in a certain sense from data or observations. A simple illustration of this is automatic recognition of handwritten digits or letters. In this case, the data consists of e.g. images of  andwritten digits and the additional information, which digit the image displays. This additional information is called label. Machine learning methods then process this data in such a way that they recognize previously unseen handwritten digits as accurate as possible. The main difference to conventional computer algorithms is that the programmers do not incorporate information about the nature of the problem at hand to the algorithm. In this case, for example, about structural differences between a handwritten "1" and "8". In this sense, a machine learning algorithm indeed learns from the data. For such learning methods mathematics plays a central role at various points: it starts with a precise definition of "learning", includes in many cases the description of the learning process and its efficient implementation as a computer algorithm, and ends with mathematically provable guarantees if and how well certain learning algorithms actually learn.

How do such guarantees look like?

This depends on many factors. Let us therefore consider the example of the so-called binary classification. Here we have only two possible labels, in the example above "1" and "8", and the learning procedure should correctly determine the label for as many future digits as possible. In the mathematical description of this learning objective, it is then assumed that the handwritten digits are in a certain sense randomly drawn from all possible images of digits, but without describing these possible images in detail. Ideally, we therefore only assume that the size of the images, e.g. 100 x 100 pixels, is known. In addition, it is assumed, that the labels can also be random to a certain extent, for example to take the natural uncertainty in scrawly handwriting into consideration.

For many learning methods it can then be proved that the optimal error rate can be achieved approximately with high probability, if sufficient data is provided to the algorithm. Moreover, the so-called "No-Free-Lunch-Theorem" shows, that for any algorithm one cannot specify the required size of the dataset without describing the random mechanism in a little more detail. Interesting guarantees, on the one hand, provide a dataset size that is as small as possible and, on the other hand, describe the random mechanism as vague as possible. Hereby, a vague description is important, since we do not want to or cannot refer to a wide knowledge on the concrete application problem when creating the learning procedure.

Finally, so-called adaptive learning methods manage to require the optimal dataset size for a large range of vague descriptions of the random mechanism without even knowing if one of the vague descriptions is valid, and if so, which one.

Which kind of special mathematical knowledge is important here?

A precise understanding of random mechanisms plays a major role in the analysis of learning methods, therefore probability theory is central. In addition, many methods cannot be understood without a detailed knowledge of certain high- or infinite-dimensional spaces. This leads to functional analysis and approximation theory. When implementing a learning process one usually has to solve an optimization problem, hence corresponding skills are important, too. Finally, a good understanding of the internal processes of a computer is essential for a concrete, own implementation of a learning process, to keep the usually high computing costs as low as possible. In this sense, one should therefore be quite broadly based.

What are current challenges?

Despite the current boom, our understanding of many learning processes is extremely limited. For example, there is no complete theory on whether neural networks, which have become very popular again in recent years, can actually learn in the above sense. In addition, most learning methods are so-called black box methods, for which it is, for example, impossible to explain a prediction that has been made. Other aspects such as privacy and fairness, which are derived from data protection and anti-discrimination grounds, play an increasingly important role, but at the moment, they are difficult to integrate into many learning processes. The same applies to the integration of specific knowledge about the problem, which after all is available in many concrete applications. Furthermore, the generation of labels is usually very costly, which leads to the demand for learning algorithms that only need a small fraction of the data to be labeled. This is of course only a small selection of open questions, but these already illustrate that there is still a considerable need for research.

What research is being conducted at the ISA in this regard?

For 20 years now, we have been dealing with the mathematics of machine learning. Initially, learning guarantees for so-called kernel-based learning methods played a central role, but since then the efficient implementation of these learning methods for large scale data has played an important role, too.  Finally, in recent years, we have also worked on learning without labels and on neural nets. In addition, we have repeatedly dealt with questions on the robustness of learning methods and the correct description of learning problems.

At the moment, we are involved in the Cluster of Excellence SimTech the International Max Planck Research School for Intelligent Systems , the Cyber Valley Initiative   of the state of Baden-Württemberg, and the European Laboratory for Learning and Intelligent Systems .

Thank you for the interview.

Prof. Ingo Steinwart
Institute of Stochastics and Application
Chair for Stochastics

To the top of the page