Negative variance components in linear mixed models: applications in plant breeding programs

Abstract

Linear mixed models (LMM) are a fundamental statistical tool in plant and animal breeding programs. They provide a flexible platform for complex data structures and appropriate modelling of (co)variances. The default setting in most LMM software is to constrain the variance component estimates to be positive, bounding them at zero to prevent negative estimates. Typically, this is done for practical reasons, such as to maintain convergence efficiency, and, due to lack of understanding of the causes for a negative value for a variance component estimate. However, there are important practical consequences which must be addressed, such as the use of incorrect denominator degrees of freedom for testing fixed effects, and a lack of understanding of the biological system which may prevent improvement in future experimental designs.
We revisit the seminal work of Nelder (1954) on the interpretation of obtaining negative variance component estimates from an analysis of variance (ANOVA) and the underpinning theoretical scenarios when a negative estimate may arise. This theory demonstrates the reason why split-plot experiments, or experiments with nested strata, are more prone to negative variance component estimates. We link this theory to residual maximum likelihood (REML) estimates obtained from LMM software, which, by definition, match the ANOVA estimates in fully balanced settings, but not if the LMM software bounds the estimates at zero.
We demonstrate the implications of constraining the variance component estimates to be positive, compared to unconstraining them, for a full diallel cross experiment in macadamia. The experiment is fully balanced and with nested strata of multiple racemes within pollination bags within trees according to a split-plot design. This experiment aims to estimate the general and specific compatibility between self-fertile and self-infertile parents, with a view to selecting compatible parental combinations. A key outcome is to inform growers about which parent combinations to plant in an orchard for maximum pollination and therefore productivity, with less reliance on pollinators, such as bees, due to the challenge of changing climates. We demonstrate that failing to appropriately model the split plot design and allowing negative variance component estimates results in incorrect significance tests, conclusions and selection decisions for the breeding programme.

Nelder (1954) The Interpretation of Negative Components of Variance. Biometrika, 41-544-548.