Inductive bias

Informally speaking, the inductive bias of a machine learning algorithm refers to additional assumptions, that the learner will use to predict correct outputs for situations that have not been encountered so far.

In machine learning one aims at the construction of algortihms, that are able to learn to predict a certain target ouput. For this the learner will be presented a limited number of training examples that demonstrate the intended relation of input and output values. After successful learning, the learner is supposed to approximate the correct output, even for examples that have not been shown during training. Without any additional assumptions, this task cannot be solved since unseen situations might have an arbitrary output value. The kind of necessary assumptions about the nature of the target function are subsumed in the term inductive bias. A classical example of an inductive bias is Occams razor, assuming that the simplest consistent hypothesis about the target function is the actually the best. Here consistent means that the hypothesis of the learner yields correct ouputs for all of the examples that have been given to the algorithm.

Approaches to a more formal definition of inductive bias are based on mathematical logic. Here, the inductive bias is a logical formula that, together with the training data, logically entails the hypothesis generated by the learner. Unfortunatelly, this strict formalism fails in many practical cases, where the inductive bias can only be given as a rough description (e.g. in the case of neural networks).