Before you start, have you read the introduction to the Ternary package?

A ternary plot can be shaded and contoured according to a known function:

library("Ternary")
par(mar = rep(0.2, 4))

FunctionToContour <- function (a, b, c) {
  a - c + (4 * a * b) + (27 * a * b * c)
}

values <- TernaryPointValues(FunctionToContour, resolution = 24L)
TernaryPlot(alab = 'a', blab = 'b', clab = 'c',
            # Place an opaque fill behind grid lines:
            panel.first = ColourTernary(values, spectrum = hcl.colors(256)))
TernaryContour(FunctionToContour, resolution = 36L)

If it is not computationally feasible to execute a function at every point, it is possible to interpolate between known values.

Inverse distance weighting is a simple approach:

# If using your own data, set 
# abc <- [Three-column matrix containing a, b, c coordinates of points]
# response <- [Vector of values at the points specified in abc]

# We'll use random data for this example.

# First we'll generate some points at which to sample:
set.seed(0)
nPts <- 50
a <- runif(nPts)
b <- runif(nPts) * (1 - a)
c <- 1 - a - b
abc <- rbind(a, b, c)

# Then we'll measure the response variable at each of those points:
response <- FunctionToContour(a, b, c)


# Now we must start a plot, to define the coordinate system
par(mar = rep(0.2, 4))
TernaryPlot(alab = 'a', blab = 'b', clab = 'c')

# Convert measured points to XY
xy <- TernaryToXY(abc)

# Use an inverse distance weighting to interpolate between measured points
Predict <- function (predXY) {
  Distance <- function (a, b) {
    apply(a, 2, function (pt) sqrt(colSums((pt - b) ^ 2)))
  }
  dists <- Distance(xy, predXY)
  id <- 1 / dists
  idw <- id / rowSums(id)

  # Return:
  colSums(response * t(idw))
}

# Predict at triangle centres
tri <- TriangleCentres(resolution = 12L) 
# Adjust the resolution to suit your own dataset

# Now we interpolate between our known values to generate a colour for each
# of our tiles
predicted <- Predict(tri[1:2, ])
map <- rbind(x = tri['x', ], y = tri['y', ], z = predicted,
             down = tri['triDown', ])

# Place a semitransparent colour fill over grid lines:
ColourTernary(map)

# Calculate contours
PredictABC <- function (a, b, c) Predict(TernaryToXY(rbind(a, b, c)))
TernaryContour(PredictABC, resolution = 36L)

# Mark the points at which we took measurements
TernaryPoints(abc, pch = 3, col = '#cc3333')

More sophisticated interpolation approaches are possible, e.g.  kriging; these may help to alleviate artefacts of sampling intensity as observed towards the bottom of the example plot above.

Interpolation within a sampled region

We might also encounter a situation where we have only sampled a small region of ternary space and do not wish to extrapolate beyond this region.

# Select some points at which to sample
set.seed(0)
nPts <- 50
a <- runif(nPts, 0.3, 0.7)
b <- 0.15 + runif(nPts, 0, 0.7 - a)
c <- 1 - a - b
abc <- rbind(a, b, c)

# Measure the response variable at each of those points:
response <- FunctionToContour(a, b, c)

# Convert measured points to XY
xy <- TernaryToXY(abc, direction = 1)

# Start a plot, to define the coordinate system
par(mar = rep(0.2, 4))
TernaryPlot(alab = "a", blab = "b", clab = "c",
            # Only plot the sampled region:
            xlim = range(xy["x", ]), ylim = range(xy["y", ])
)

# Get centres of tiles to plot
tri <- TriangleCentres(resolution = 36L)
# Adjust the resolution to suit your own dataset

# Calculate which triangular tiles are within the boundaries of our data
inHull <- TriangleInHull(tri, abc)

# Pre-generate an empty list of predictions
predicted <- rep(NA, ncol(tri))

# Now we interpolate between our known values to generate a colour for each
# of our tiles
predicted[inHull$inside] <- Predict(tri[1:2, inHull$inside])
map <- rbind(tri, z = predicted)

# Colour only those tiles where we have made a prediction
ColourTernary(map)

# Calculate contours
PredictABC <- function (a, b, c) Predict(TernaryToXY(rbind(a, b, c)))
TernaryContour(PredictABC, resolution = 36L,
               # `within` controls where contours are calculated
               within = inHull$hull)


# Mark the points at which we took measurements
TernaryPoints(abc, pch = 3, col = "#cc3333")

# Mark the region in which we sampled
polygon(t(xy)[chull(t(xy)), ], border = "#cc3333")