Coresets, Sparse Greedy Approximation, and the Frank-Wolfe Algorithm

Ken Clarkson

The problem

Find `x_**` maximizing `f(x)`, for `x in S`, where:

`f(x)` is a concave function
`S` is the simplex `{x | sum_i x_i = 1, x_i >= 0}`

Vertices of `S` are points `e(i)`, where `e(i)` is the vector with `e(i)_i = 1`, others zero
Points in `S` are the ones that give you convex combinations

The approximation algorithm

Compute a sequence `x_{:(0):}, x_{:(1):}, x_{:(2):},...` as follows

Let `x_{:(0):} := ` the vertex of `S` with largest `f` value
For `k=0,1,2,ldots` :

Find index `i'` of largest coordinate of the gradient `grad f{:(x:}_{:(k):}{:):}`
Let `x_{:(k+1):} := ` point on segment from `x_{:(k):}` to `e(i')` maximizing `f(x)`
That is, find `alpha' in [0,1]` maximizing `f(x_{:(k):} + alpha(e(i') - x_{:(k):}))`

What the algorithm is doing

At `x := x_{:(k):}`,

`f(y) approx f(x) + (y-x)^T grad f(x)`

What is the maximum of that linear approximation?

`max_{y in S} y^T grad f(x) = max_i grad f(x)_i = grad f(x)_{i'}`

...with `y=e(i')`. So
`max_{y in S} f(x) + (y-x)^T grad f(x) = max_i grad f(x)_i + f(x) - x^T grad f(x)`
The algorithm finds the vertex `e(i')` maximizing the linear approximation, and moves toward it.

Example: Minimum Enclosing Ball

The problem is,

Given: a set `P={p_1,ldots,p_n}` of points
Find: their minimum enclosing ball

Also known as the 1-center, smallest enclosing sphere...

Algorithm, as applied to MEB

Let:

`b in RR^n` have `b_i := p_i^2 := p_i^Tp_i`;
`A` be a matrix whose columns are the `p_i`;

Then:

For `f(x) := x^Tb - (Ax)^2`, the MEB problem is dual to `max_{x in S} f(x)`
That is, max problem gives a lower bound for MEB, and `c_** := Ax_**` is the center of the MEB

For `i' := argmax_i grad f(x)_i` chosen by the algorithm,

`grad f(x) = b - 2A^TAx`, so
`grad f(x)_i = p_i^2 - 2p_i^TAx = p_i^2 - 2p_i^Tc = (p_i-c)^2 - c^2`

So: `p_{i'}` is the point of `P` farthest from `c=Ax`

Algorithm, as applied to MEB, cont.

Vector `x` prescribes a weighted combination of the input points
Max gradient of `f iff` farthest point from current center
Algorithm first proposed for MEB only [BC03]

Why is this algorithm interesting?

Iterates are sparse: iterate `x_{:(k):}` has at most `k+1` nonzero coordinates
Many applications
Simple
Iterates are provably good approximations

Additive error `O(1//k)`

Relation to coresets `approx` sparsity + approximation

Application areas

Given a set of points: find minimum enclosing ball, ellipsoid, axis-aligned ellipsoid;
Support vector machines (SVM)

hard margin, `L_2`-SVM, `L_2`-SVR

"Boosting", for example, Adaboost

`approx` finding best convex combinations of classifiers

Convex approximation

of a point by other points `f(x) := -||p-Ax||^2`
of a function by other functions

density mixtures

w.r.t. different norms

`L_p` distance instead of `L_2`
KL and other divergences [AS]

Hard margin SVM

Training problem is: given red and blue points, find the thickest slab that separates them
Corresponding `f(x)` is a quadratic function
Points likely to be in very high-dimensional "feature space"

Simplicity

Convergence is slow: additive `epsilon` in `O(1//\epsilon)` steps

Gradient ascent has "linear convergence": `O(log(1//epsilon))` steps
Newton-type methods have "quadratic convergence": `O(log log (1//\epsilon))` steps

However, for Newton:

Fewer steps, but much more work and space per step

Solve linear system each step

For gradient ascent, constant may be large
Simple may be better for large-scale problems [TKS][H][Nemi]

"Good enough approximation"

Sometimes answer need only be "good enough", e.g., estimation/learning
Statistical estimators: fit parametric model to noisy data

e.g., fit a line to points

Noise in data `=>` exact fit to data not helpful
Error = Data error + Opt. error

No reason to reduce Opt. error to zero

Why is this algorithm interesting?

Iterates are sparse: iterate `x_{:(k):}` has at most `k+1` nonzero coordinates
Many applications
Simple
Iterates are provably good approximations

Additive error `O(1//k)`

Relation to coresets `approx` sparsity + approximation

Aside: history

A variant is called "sparse greedy approximation" in machine learning [Z03]

Tries every `i'`, not just the one realizing max `grad f(x)`;
Boosting, density mixtures, divergences studied there
Showed convergence results

Variants for some special cases are called "coreset-based algorithms" in computational geometry [BHI02]

Common variant works harder: finds opt `x` in given `k`-face
Applied to MEB, other enclosure problems, SVM, shape approximation, `k`-centers...

Algorithm is a special case of the Frank-Wolfe algorithm [FW56]

General convergence results shown

Approximation properties

For many of the `f(x)`, there is a "nonlinearity measure" `C_f` so that

`f{:(x:}_{:(k):}{:):} >= f(x_**) - C_f/(k+3)`

`C_f` is related to norm of second derivative of `f`

The flatter the graph of `f` is, the smaller `C_f` is
When `f(x)` is linear, `C_{:f:} = 0` and optimum is very sparse: it's a vertex of `S`

The Wolfe Dual

Minimize, for `x in RR^n` and with `z(x) := max_i grad f(x)_i`,

`w(x) := z(x) + f(x) - x^T grad f(x)`

(Use Lagrangian relaxation, use KKT to get `lambda = ze - grad f(x)`, substitute for `lambda`)
Recall `w(x)` is optimum within `S` of linear approx. at `x`

For `x in S`, `w(x) >= w(x_**) = f(x_**) >= f(x)`
Gap `w(x) - f(x) = z(x) - x^T grad f(x) = (e(i') - x)^T grad f(x)`

Coordinate `i'` as used in algorithm

Example dual: MEB

Used
`f(x):= x^Tb - x^TA^TAx = x^Tb - c^2`
for MEB, where `c=Ax`
So

`z(x) + f(x) - x^T grad f(x)`	`= z(x) + (x^Tb - c^2) - x^T(b-2A^Tc)
	` = z(x) + c^2`

Recalling `z(x) = max_i grad f(x)_i = (p_{i'}-c)^2 - c^2`
So `z(x) + c^2` is the max squared distance of `c` to any `p_i`

One step improvement

For `y := x + alpha(e(i') - x)`, Taylor's theorem says:
`qquad f(y) approx f(x) + alpha(e(i')-x)^T grad f(x)`
`qquad qquad qquad + alpha^2(e(i')-x)^T grad^2 f(x) (e(i')-x)`

By concavity of `f`, first two terms bound `f(x)` from above
Choose `C_f` to be smallest value
so that `f(y) >= f(x)+ alpha(e(i')-x)^T grad f(x) - alpha^2 C_f`
But: `(e(i')-x)^T grad f(x) = w(x) - f(x)`
So: `f(y) >= f(x) + alpha(w(x) - f(x)) - alpha^2 C_f`

One step improvement, cont.

Have: `f(y) >= f(x) + alpha(w(x) - f(x)) - alpha^2 C_f`
So if `w(x) - f(x) gg 0`, then `f(y) gg f(x)`
Let `h(x) := (f(x_**)-f(x))//4C_f`, `g(x) := (w(x) - f(x))//4C_f`

Since `w(x) >= f(x_**) >= f(x)`, have `g(x) >= h(x)`

Restating, have: `h(y) le h(x) - alpha g(x) + alpha^2//4`
Choosing best `alpha` gives `h(y) le h(x) - g(x)^2 le h(x) - h(x)^2`
Have `h{:( :}x_{:(k+1):}{:):} leq h{:( :}x_{:(k):}{:):} - h{:( :}x_{:(k):}{:):}^2 => h{:( :}x_{:(k):}{:):} le 1//(k+3)`

What is `C_f`?

`C_{:f:} le` sup of `-{:1/2:}(y-x)^T grad^2 f({:bar x:})(y-x)`, over `{:bar x:}, x,y in S`
Main case: `f(x)` has the form `{:hat f:}(Ax)`

That is, optimizing `hat f` within the polytope `AS`
`grad^2 f(x) = A^T grad^2{:hat f:}(Ax)A`
`C_{:f:} le` sup of `-{:1/2:}(y-x)^T A^T grad^2 {:hat f:}(A{:bar x:})A(y-x)`
Quadratic `f` has this form, and quadratic term of `{:hat f:}` is `-c^2`

`C_{:f:} le` sup of `(Ay - Ax)^2`, or `diam(AS)^2`

Example `C_f`: MEB and SVM

For MEB, `C_{:f:} le 4R^2 = 4f(x_**)`, where `R` is the MEB radius
So additive error `epsilon C_f` becomes relative error `4epsilon`
Same bound on `C_f` for any quadratic problem

For SVM, relative error depends on `rho_**^2//R^2`, where `rho_**` is opt. thickness
The VC dimension of set of regions that are radius-`R` balls with thickness-`rho` central slice removed is `R^2//rho_**^2`

Variations

Have: `h(y) le h(x) - alpha g(x) + alpha^2//4`
In particular, `h(y) le h(x) - g(x)^2
Lazy variant:

Don't find best `alpha'`, use `alpha_k := 2//(k+3)`
Get same guarantees

Primal/dual approximation:

If `g{:( :}x_{:(k):}{:):}` is always big, eventually `h(x) lt 0`
Implies, eventually see `g{:( :}x_{:(k):}{:):} lt epsilon` and `h{:( :}x_{:(k):}{:):} lt epsilon`

After `1//epsilon` steps, `h() le epsilon`; after another `1//epsilon`, some `g() le epsilon`

For MEB: `x` proves that there is a small ball, and that no ball can be much smaller

Primal/dual and coresets

So: some `hat x` is good in primal, dual, and has `O(1//epsilon)` nonzero entries
Nonzero coordinates `N subset {1 ldots n}`, and `|N| = O(1//epsilon)`
`N` is almost a combinatorial specification of a good approx. solution

For MEB, `N harr` a subset of `P` that specifies a good approximate solution
But: need values of `hat x` at those coordinates
Use canonical `x` for `N`: the best `x` with those nonzeros `=:x_N`

Cannot simply run algorithm, and use `N` for resulting `x`: `f(x_N) >= f(x)`, but no grip on `w(x_N)`

Coresets

Another variation:

Pick `i'` the same way, but make `x_{:(k+1):}` maximize `f(x)` over all `x` with same nonzeros
That is: `N := N cup {i'}`, and find `x_{N}`
The measure `h()` decreases as fast, and as before, eventually `g() lt epsilon`

So the algorithm is:

Let `N :=` index of the vertex of `S` with largest `f` value
While `g(x_N) le epsilon`:

Find index `i'` of largest coordinate of the gradient `grad f(x_N)`
Let `N := N cup {i'}`

Why are coresets interesting?

An approach for densest-ball problem:

Given points `P`, find `P' subset P` with `|P'| > n//2` and `P'` in smallest ball over all such subsets
Approximation algorithm:

Try all subsets of size `O(1//epsilon)`, one of them is coreset for MEB of `P'`

Implications for sample complexity of hard-margin SVM
Imply algorithms for:

`k`-center, `k`-median, cylinder-fitting, `k`-means

Running times depend on `epsilon`, but often, independent of dimension

Important for SVM, in particular

Coreset variants

"Away" steps: work even harder, for a smaller core-set

After getting `x_{:N:}` with `|N|=K+1` nonzeros, at each iteration, also:

Pick the coordinate `i''` at which `x_N` is smallest over all nonzero coords, and
Set `N:= N setminus {i''}`
Still make progress in `f`, for `K` a small multiple of `1//epsilon`, and gap `g` large

Allows smaller (near-optimal) constant in the core-set size

Be lazier

Use a fixed schedule of stepsize `alpha_k`,
No change in provable bounds

Probabilistic coreset existence

R.V. `Y`: pick `i` to be `1` with probability `x_i^**`
Pick `Y_j` for `j=1...K`, and `sum_j Y_j//K` is a good sparse approx. solution
Additive error `sum_i x_i^** p_i^2//r`, for quadratic `{:hat f:}(Ax)`, columns of `A` are `p_i`
This is no big improvement for MEB, but is in another case
Method of cond. prob.?

Concluding remarks

A simple, and old, algorithm has many applications and implications, even today
New things:

General analysis for some approximation algorithms;
General proof of coreset existence, including with "away" steps
Faster algorithm for many learning applications
Sharper bounds for SVM coreset constants (and constants matter)
General clue about where sparsity comes from

Further implications for sample complexity?
Good relative error for MEB: `C_{:f:} approx` OPT; good relative bounds for other problems?
Distant relative of algorithms for finding best linear approximation ([GMS,Tropp]); what implications here?

Bonus Animation: MEB Approximation

Move toward midpoint of two recent farthest points

Core-sets, SGA, Frank-Wolfe

TOC talk

Coresets, Sparse Greedy Approximation, and the Frank-Wolfe Algorithm

Ken Clarkson

The problem

The approximation algorithm

What the algorithm is doing

Example: Minimum Enclosing Ball

Algorithm, as applied to MEB

Algorithm, as applied to MEB, cont.

Why is this algorithm interesting?

Application areas

Hard margin SVM

Simplicity

"Good enough approximation"

Why is this algorithm interesting?

Aside: history

Approximation properties

The Wolfe Dual

Example dual: MEB

One step improvement

One step improvement, cont.

What is `C_f`?

Example `C_f`: MEB and SVM

Variations

Primal/dual and coresets

Coresets

Why are coresets interesting?

Coreset variants

Probabilistic coreset existence

Concluding remarks

Bonus Animation: MEB Approximation