Linear regression operator - 'greedy' feature selection option

amitd

Can someone please help with more details on how the "greedy" feature selection inLinear Regressionoperator works?

In theOptimize Selectionoperator, the two greedy algorithms (forward selection and backward elimination) are clearly specified. However, in the case of theLinear Regressionoperator, it is not yet clear which algorithm/approach is used in case of the built-in "greedy" feature selection dropdown option for theLinear Regressionoperator.

I could not find any elaboration on this in thedocumentation for Linear Regressionoperator. Also, I checked thesource codefor this option. According to it -

This class implements an internal forward selection for the linear regression. It uses the Akaike Criterion that is maximized roundwise. Each round the attribute minimizing the akaike criterion is deselected.

I am trying to understand the exact model selection process happening here and the description and the code is a bit unclear to me.

1. The above statement says internal forward selection. But the later sentence "..attribute minimizing the akaike criterion is deselected." suggests backward elimination. Which one is it? Can someone elaborate on this?

2. How is AIC criterion computed here? A couple of sources (link1andlink2) suggest AIC for linear regression as n x ln(SSE / n) + 2 (k + 1) where n is the number of observations and k is the number of predictors (one is added for the intercept term). The source code states -

akaike = (numberOfExamples - numberOfUsedAttributes) + 2 * numberOfUsedAttributes;

这有点令人困惑的well. Any insights would be appreciated.

jacobcybulski

When you peek into the operator's source code (yes everybody can do so), you will find the following (slightly cryptic) comment: "This class implements an internal forward selection for the linear regression. It uses the Akaike Criterion that is maximized roundwise. Each round the attribute minimizing the akaike criterion is deselected."

jacobcybulski

What it means is that it maximises AIC, it considers all attributes as candidates for addition and removes one candidate at a time, based on the lowest overall AIC when they are added (sort of as a test), iteratively! AIC is a relative criterion, so it can only be used to compare one model vs the other - so we will not know AIC of the model with an added attribute until it is added!