The aim is to construct the discriminant function in order to separate two populations. For the sake of brevity we will keep ourselves on the example of the linearly separable populations, and thus, the usage of the Linear Discriminant Function (LDF) (find out more about LDF here , or here ).
Let's assume that we have already generated the set of features to be used by our LDF, in order to distinguish objects from two different populations, and that we have been smart enough to choose the best possible constellation of features for our set (given that we had made some measurements, testing of different sets and combinations of features, etc.).The next step is to try to improve the performance of our LDF, and hence, we are to examine more features to be concatenated to the basic set (of ''best'' features). So, we can make an assumption that features that were not primarily included are somehow weaker regarding their discriminating power, meaning: they really do tell less about objects' characteristics which makes objects belonging to different classes classifiable. The question that arises at this point is:
Should we include more features, and how can we know if new features will increase the discriminating power of our LDF? Furthermore, it would be the best if we could find the way to estimate the performance of LDF regarding each possible set of features we are to use, and somehow avoid computing the actual function.
So, let's handle this problem...