The FAMoS package comes with a variety of functions that allow to plot the outcome of the model selection algorithm in a convenient way. These functions can be helpful to assess the performance of the model selection algorithm and help in interpreting the support of each model parameter.

library(FAMoS)
 #set the future plan (see future documentation for more information)
 future::plan(future::sequential)

 #create data with standard deviation of 1
 x.values <- 1:7
 y.values <-  9 * x.values^2 - exp(2 * x.values)
 sd.y.values <- rep(1,7)

 #define initial parameter values
 inits <- c(p1 = 3, p2 = 4, p3 = -2, p4 = 2, p5 = 0)

 #define cost function that returns the negative log-likelihood
 cost_function <- function(parms, x.vals, y.vals, sd.y){
  # restrict the search range to -5 to +5
  if(max(abs(parms)) > 5){
    return(NA)
  }
  with(as.list(c(parms)), {
    res <- p1*4 + p2*x.vals + p3^2*x.vals^2 + p4*sin(x.vals)  - exp(p5*x.vals)
    diff <- sum((res - y.vals)^2/sd.y)
  })
 }

 #define swap parameter sets
 swaps <- list(c("p1", "p5"))
 
 #perform model selection
 res <- famos(init.par = inits,
            fit.fn = cost_function,
            nr.of.data = length(y.values),
            homedir = getwd(),
            init.model.type = c("p1", "p2"),
            optim.runs = 1,
            method = "swap",
            swap.parameters = swaps,
            x.vals = x.values,
            y.vals = y.values,
            sd.y = sd.y.values,
            save.performance = F)

To assess the performance of a specific FAMoS run, the function famos.performance can be used. This generates a plot showing the improvement in the information criterion over the course of each FAMoS iteration along with the corresponding best model.

famos.performance(input = res$mrun)

The circles in the lower plot what happened during each FAMoS iteration. Here, parameter p1 was swapped for p5 in the second iteration (blue circles), parameter p3 was added in the third iteration (green circle) and parameter p2 was removed in the fifth iteration (red circle). After that, FAMoS tried to find better models with the succession of the methods backward, forward and swap, but did not succeed. Therefore the model selection procedure was terminated after the eighth iteration. The best model, therefore consists of the parameters p3 and p5 and the corresponding AICc is shown on the right axis of the upper plot.

Note that the performance of FAMoS can also be checked while the algorithm is still running. After the third iteration, a plot generated by famos.performance will be stored in the "FAMoS-Results/Figures/" folder. This plot will be updated at the end of each iteration. Alternatively, the function can also be called in the console.

In a next step, we can check how the tested models are distributed in terms of the used information criterion (here: AICc). To this end, we call the function ic.order.

 ic.order(input = getwd(), mrun = res$mrun, log = "y")

We can also use this function to check how certain parameters affect the value of the information criterion

 par(mfrow = c(1,2))
 ic.order(input = getwd(), mrun = res$mrun, colour.par = "p1", log = "y")

Here, we can see that all models that contain p5 perform much better than models that don't have this parameter.

We can analyse the support of each parameter even more closely by calculating the Akaike weights with the function aicc.weights, which gives a normalised description of the importance of parameters. Roughly, values close to 1 indicate high and values close to 0 weak to no support. Parameters with intermediate support might indicate a necessity of having a certain feature in the model to describe the data, but this feature might be achieved by different parameters.

aicc.weights(input = getwd(), mrun = res$mrun)

Plotting the results of the model selection process