September 27, 2016

## Course Logistics

• Office hour: 2-3pm on Tuesdays or by appointment
• New office: 4623 SPH-I (within Suite 4605)
• Homework 1 due 11:59pm on October 10, 2016 (Questions?)

## Inference Problems: Three Types of Tasks

• Marginal probabilities. Compute marginals of variables (given model parameters $$\mathbf{\theta}$$): $$p(x_i\mid \mathbf{\theta})=\sum_{\mathbf{x}': x_i'=x_i}p(\mathbf{x}'\mid \mathbf{\theta}).$$ (Or conditional probabilities: posterior distribution)

• Partition function. For a Gibbs distribution representable as normalized products of factors: $$p(\mathbf{x})=\frac{1}{Z}\prod_{C\in\mathcal{C}}\psi_{C}(\mathbf{x}_C)$$ with respect to a graph $$\mathcal{G}$$ with cliques $$\mathcal{C}$$, compute $Z(\mathbf{\theta})=\sum_{\mathbf{x}}\prod_{C\in\mathcal{C}}\psi_{C}(\mathbf{x}_C)$

• Maximum A Posteriori (MAP) Inference. Compute the variable configuration with the hightest probability: $$\hat{\mathbf{x}}=\arg \max_{\mathbf{x}}p(\mathbf{x}\mid \mathbf{\theta}),$$ where $$p$$ is a generic notation for joint or conditional distributions with unobserved variables $$\mathbf{X}=\mathbf{x}$$.

## Compute Marginal Probabilities

• Let $$\mathbf{X}=\{X_1,\ldots,X_d\}$$ denote the set of variable nodes, each of which can range from $$1$$ to $$k$$; $$F=\{\psi_{\alpha_1},\ldots, \psi_{\alpha_s}\}$$ denote the set of factor nodes, where $$\alpha_1, \ldots, \alpha_s \subset \{1,\ldots, d\}$$.
• Compute the marginal probability naively, we do: $p(X_1=x_1)=\frac{1}{Z}\sum_{x_2,\ldots, x_d}\prod_{i=1}^s \psi_{\alpha_i}(\mathbf{x}_{\alpha_i})$

• Computational complexity: $$\mathcal{O}(k^{d-1})$$ additions of values read from the probability table, one row for each configuration $$X_2=x_2,\ldots,X_d=x_d$$.

• Instead, we choose an ordering $$\mathcal{I}$$ among $$\{X_1,\ldots,X_d\}$$ and utilize the factorization of $$p(\mathbf{x})$$. (Variable Elimination)

## Factor Graph

• Explicitly encondes the relations among a subset of variables, which sometimes are not possible by clique potential parameterization (see an example on the next slide).
• Supports the inference/computational techniques
• Unifies directed and undirected graphical models
• Example (whiteboard)

## Factor Graph: A Refinement

• Example:
• $$\mathcal{H}=(V,E)$$: complete (fully connected) undirected graph with vertices $$V$$ and edges $$E$$. $$|V|=d$$
• $$P$$: positive distribution over $$d$$ binary nodes; has only pairwise potentials (pairwise dependencies cannot be conditioned away).
• $$P$$ is Markov to $$\mathcal{H}$$, but not Markov to any graph missing even one edge.
• The graph $$\mathcal{H}$$ can only guide use to use the maximal clique potential parametrization. For a omplete graph, we use a single, large potential $$\psi(X_1,\ldots,X_d)$$, which requires $$2^d$$ parameters.
• However, $$P$$ by definition can be represented in a Gibbs distribution only with pairwise potentials $$\psi_{{j}{j'}}(X_j,X_{j'})$$, which requires $$4{d \choose 2}$$ parameters
• When $$d=10$$, $$2^d=1,024$$, much bigger than $$4{d \choose 2}=180$$.

## Variable Eliminations for $$p(X_1=x_1)$$ for $$p(X_1,\ldots, X_d)$$

1. Initialize an active list of potentials as read off from the factor graph.
2. Choose an ordering $$\mathcal{I}$$ for the variables in which $$x_1$$ appears last
3. For each $$i=1,\ldots,d-1$$, let $$\mathcal{I}_i=j$$, do the following:
• Find all potentials in the currently active list that reference $$x_j$$ and remove them from the active list
• Let $$\phi_{j}(X_{T_j})$$ denote the product of these potentials, with $$T_j$$ being the union of nodes appearing in the active potentials referencing $$X_j$$.
• Calculate $$\tau_i(x_{T_j-X_j})=\sum_{x_j}\phi_{j}(x_{T_j})$$
• Place $$\tau_i(x_{T_j-X_j})$$ on the active list as one potential function
4. Finally, we obtain $$p(x_1)=\frac{1}{Z}\tau_{d-1}(x_1)$$

## Example: Variable Elimination for Factor Graph

• Let $$X_i\in \{1,\ldots,k\}$$.
• Describe how you would compute $$p(X_1=x_1)$$.
• Computational complexity of variable elimination: $$\mathcal{O}((d-1)k^r)$$, where $$d$$ is the number of variables, $$k$$ is the maximum value a variable can take and $$r$$ is the number of variables participating in the largest intermediate "factor".
• Finding a good ordering can reduce the computational complexity.

## Comment

• Next Lecture: Belief propagation (Sum-Product Algorithm on Polytrees)

• Required reading for the week: Chapter 9. Koller and Friedman (2009)