What is a Mixture Model?

A mixture model is a probabilistic model that represents a population consisting of multiple subpopulations, where each observation is assumed to come from one of several component distributions. Mixture models are particularly useful for modeling data that exhibits multimodal behavior or comes from heterogeneous sources.

Formally, a mixture model with $K$ components is defined as:

\[f_{mix}(x; \Theta, \pi) = \sum_{k=1}^K \pi_k f_k(x; \theta_k)\]

Where:

• $f_k(x; \theta_k)$ is the $k$-th component distribution with parameters $\theta_k$
• $\pi_k$ is the mixing coefficient (or mixing weight) for component $k$
• $\Theta = \{\theta_1, \ldots, \theta_K\}$ are the component parameters
• $\pi = \{\pi_1, \ldots, \pi_K\}$ are the mixing coefficients with $\sum_{k=1}^K \pi_k = 1$ and $\pi_k \geq 0$

Generative Process

The generative process for a mixture model can be described as follows:

\[\begin{align*} z_i &\sim \text{Cat}(\pi) \\ x_i &\mid z_i = k \sim f_k(x; \theta_k) \end{align*}\]

Where:

• $z_i$ is a latent assignment variable indicating which component generated observation $x_i$
• $z_i = k$ means observation $x_i$ was generated by component $k$
• $\text{Cat}(\pi)$ is the categorical distribution with probabilities $\pi$

This two-step process first selects a component according to the mixing weights, then generates an observation from that component's distribution.

Key Properties

Identifiability: Mixture models are generally identifiable up to permutation of the component labels. This means that swapping component labels and their associated parameters yields an equivalent model.

Model Selection: The number of components $K$ is typically unknown and must be determined using model selection criteria such as AIC, BIC, or cross-validation.

Latent Structure: The latent variables $z_i$ provide a natural clustering interpretation, where observations are probabilistically assigned to components.

Gaussian Mixture Model

A Gaussian Mixture Model (GMM) is a mixture model where each component is a multivariate Gaussian distribution. GMMs are among the most widely used mixture models due to their mathematical tractability and flexibility in modeling continuous data.

GaussianMixtureModel

GaussianMixtureModel(k::Int, data_dim::Int)

Mathematical Formulation

For a GMM with $K$ components in $D$ dimensions, the mixture density is:

\[f_{GMM}(x; \Theta, \pi) = \sum_{k=1}^K \pi_k \mathcal{N}(x; \mu_k, \Sigma_k)\]

Where each component is a multivariate Gaussian:

\[\mathcal{N}(x; \mu_k, \Sigma_k) = \frac{1}{(2\pi)^{D/2} |\Sigma_k|^{1/2}} \exp\left(-\frac{1}{2}(x - \mu_k)^T \Sigma_k^{-1} (x - \mu_k)\right)\]

The parameters for each component $k$ are:

• $\mu_k \in \mathbb{R}^D$: the mean vector
• $\Sigma_k \in \mathbb{R}^{D \times D}$: the covariance matrix (positive definite)

Applications

GMMs are particularly effective for:

• Density estimation for continuous multimodal data
• Soft clustering where observations can belong to multiple clusters with different probabilities
• Dimensionality reduction when combined with factor analysis
• Anomaly detection by identifying low-probability regions

Poisson Mixture Model

A Poisson Mixture Model (PMM) is designed for modeling count data that exhibits overdispersion or multimodality. Each component follows a Poisson distribution, making it suitable for discrete, non-negative integer observations.

PoissonMixtureModel

PoissonMixtureModel(k::Int)

Mathematical Formulation

For a PMM with $K$ components, the mixture density is:

\[f_{PMM}(x; \lambda, \pi) = \sum_{k=1}^K \pi_k \text{Poisson}(x; \lambda_k)\]

Where each component is a Poisson distribution:

\[\text{Poisson}(x; \lambda_k) = \frac{\lambda_k^x e^{-\lambda_k}}{x!}\]

The parameter for each component $k$ is:

• \[\lambda_k > 0\]

  : the rate parameter (both mean and variance of the Poisson distribution)

Applications

PMMs are commonly used for:

• Count data modeling with heterogeneous populations
• Modeling overdispersed count data where variance exceeds the mean
• Analyzing arrival processes with multiple underlying rates
• Biological applications such as modeling gene expression counts

Learning in Mixture Models

StateSpaceDynamics.jl implements the Expectation-Maximization (EM) algorithm for parameter estimation in mixture models. EM is an iterative algorithm that finds maximum likelihood estimates in the presence of latent variables.

fit!(gmm::GaussianMixtureModel, data::AbstractMatrix{<:Real}; <keyword arguments>)

fit!(
    pmm::PoissonMixtureModel, data::AbstractMatrix{<:Integer};
    maxiter::Int=50, tol::Float64=1e-3, initialize_kmeans::Bool=false,
)

Expectation-Maximization Algorithm

The EM algorithm alternates between two steps:

Expectation Step (E-step)

Calculate the posterior probabilities (responsibilities) that each observation belongs to each component:

\[\gamma_{ik} = p(z_i = k \mid x_i, \theta^{(t)}) = \frac{\pi_k^{(t)} f_k(x_i; \theta_k^{(t)})}{\sum_{j=1}^K \pi_j^{(t)} f_j(x_i; \theta_j^{(t)})}\]

Where $\gamma_{ik}$ represents the responsibility of component $k$ for observation $x_i$.

Maximization Step (M-step)

Update the parameters by maximizing the expected complete data log-likelihood:

Mixing coefficients:

\[\pi_k^{(t+1)} = \frac{1}{N} \sum_{i=1}^N \gamma_{ik}\]

For Gaussian components:

\[\begin{align*} \mu_k^{(t+1)} &= \frac{\sum_{i=1}^N \gamma_{ik} x_i}{\sum_{i=1}^N \gamma_{ik}} \\ \Sigma_k^{(t+1)} &= \frac{\sum_{i=1}^N \gamma_{ik} (x_i - \mu_k^{(t+1)})(x_i - \mu_k^{(t+1)})^T}{\sum_{i=1}^N \gamma_{ik}} \end{align*}\]

For Poisson components:

\[\lambda_k^{(t+1)} = \frac{\sum_{i=1}^N \gamma_{ik} x_i}{\sum_{i=1}^N \gamma_{ik}}\]

Convergence and Initialization

The EM algorithm is guaranteed to converge to a local maximum of the likelihood function. However, the quality of the solution depends heavily on initialization:

• Random initialization: Parameters are randomly initialized
• K-means initialization: Use K-means clustering to initialize component means and mixing weights
• Multiple random starts: Run EM from multiple random initializations and select the best solution

Model Evaluation and Selection

Log-Likelihood

The log-likelihood of the data under the fitted model provides a measure of model fit:

loglikelihood(gmm::GaussianMixtureModel, data::AbstractMatrix{<:Real})

loglikelihood(pmm::PoissonMixtureModel, data::AbstractMatrix{<:Integer})

\[\ell(\Theta, \pi) = \sum_{i=1}^N \log \left( \sum_{k=1}^K \pi_k f_k(x_i; \theta_k) \right)\]

Information Criteria

For model selection (choosing the optimal number of components $K$):

Akaike Information Criterion (AIC):

\[\text{AIC} = -2\ell(\hat{\Theta}, \hat{\pi}) + 2p\]

Bayesian Information Criterion (BIC):

\[\text{BIC} = -2\ell(\hat{\Theta}, \hat{\pi}) + p \log(N)\]

Where $p$ is the number of free parameters in the model.

Inference and Applications

Posterior Probabilities

After fitting, the posterior probability that observation $x_i$ belongs to component $k$ is:

\[p(z_i = k \mid x_i, \hat{\Theta}, \hat{\pi}) = \frac{\hat{\pi}_k f_k(x_i; \hat{\theta}_k)}{\sum_{j=1}^K \hat{\pi}_j f_j(x_i; \hat{\theta}_j)}\]

Hard Assignment

For clustering applications, observations can be assigned to their most likely component:

\[\hat{z}_i = \arg\max_{k} p(z_i = k \mid x_i, \hat{\Theta}, \hat{\pi})\]

Sampling

New observations can be generated from the fitted mixture model by:

1. Sampling a component $k \sim \text{Cat}(\hat{\pi})$
2. Sampling from that component: $x \sim f_k(x; \hat{\theta}_k)$

Practical Considerations

Computational Complexity

• Time complexity: $O(NKD \cdot \text{iterations})$ where $N$ is the number of observations, $K$ is the number of components, and $D$ is the dimensionality
• Space complexity: $O(NK + KD^2)$ for storing responsibilities and covariance matrices (GMM)

Common Issues

Singular covariance matrices: In GMMs, components may collapse to single points, leading to singular covariance matrices. Regularization techniques include adding a small value to the diagonal.

Empty components: Some components may receive very little probability mass during EM iterations. This can be addressed by reinitializing empty components.

Convergence to local optima: EM is sensitive to initialization. Multiple random starts and careful initialization strategies are recommended.

Reference

For comprehensive mathematical foundations and advanced topics in mixture models, we recommend:

• Pattern Recognition and Machine Learning, Chapter 9 by Christopher Bishop
• The Elements of Statistical Learning, Chapter 6 by Hastie, Tibshirani, and Friedman
• Finite Mixture Models by Geoffrey McLachlan and David Peel