CV Chapter Chapter 8 Motion and Optical Flow

1. Sometimes motion is foremost cure

2. Even “impoverished” motion can evoke percept

1. Estimate 3D structure

2. Segmenting based on motion cues

3. Learning dynamical models

4. Recognizing events and activities

5. Motion stabilization for better video quality

1. Directly from optic flow

2. Indirectly to create correspondence for SfM

Projection of 3D scene motion into image

1. Position P(t)

2. Velocity V = dP/dt = [V_x,V_y,V_z]

1. Position p(t) = (x(t),y(t)) = f * P/Z

2. Velocity v=f*(ZV-V_zP)/Z²=(fV-V_zp)/Z with respect to v_x=dx/dt and v_y=dy/dt

1. V is constant

2. Velocity v=1/Z*(v₀-V_zp) with v₀=(fV_x,fV_y) with respect to v_x=fV_x-V_zx/Z and v_y=fV_y-V_zy/Z

Every motion vector points toward or away from v₀

Apparent motion of brightness patterns in the image

1. Ideally the same

2. Can be caused by lighting changes without any actual motion

3. Imagine rotating sphere under fixed lighting vs fixed sphere under moving illumination

1. Brightness constancy

2. Small motion of points

3. Spatial coherence with neighbors of point have similar (u,v)

1. One point at different positions has same brightness

2. I(x,y,t-1)=I(x+u(x,y),y+v(x,y),t)

3. I(x,y,t-1)~I(x,y,t)+I_x*u(x,y)+I_y*v(x,y)

4. I_x*u+I_y*v+I_t~ or =0 with I_x,I_y spatial and I_t temporal derivative

5. GradI*(u,v)+I_t=0

Given I(x,y,t-1) and I(x,y,t) estimate apparent motion field u(x,y) and v(x,y) between the two subsequent frames

Perceived is different to actual motion due to small view

Use 5² window yielding 25 equations with least squares problem (LSP) [I_x(p₁),I_y(p₁);…;I_x(p₂₅),I_y(p₂₅)]*[u;v]=-[I_t(p₁);…;I_t(p₂₅)]

Solution of (A^TA)*d=A^T*b the so-called Lucas-Kanade equation (LKE)

A^TA=[SumI_x²,SumI_xI_y;SumI_xI_y,SumI_y²]=Sum[I_x;I_y][I_x,I_y]=Sum GrdI*GrdI^T

A^T*b=[SumI_xI_t;SumI_yI_t]

If (u,v) satisfies LKE

1. Should be invertible

2. Entries not to small (due to noise)

3. Well conditioned

1. Similar to Harris corner detector second moment matrix

2. Eigenvectors relate to edge direction and magnitude

3. Similar eigenvalues indicate low or high texture region

4. Different eigenvalues indicate edge

Indicate moving up with rotating

1. Iterative refinement (IR)

2. Coarse-to-fine optical flow estimation (CTFE)

1. Estimate velocity at each pixel using iterative LKE

2. Warp image using estimated flow field // Error propagation

3. Refine estimate by repeating process

1. Run iterative LKE

2. Warp and upsample

3. Repeat process

1. Reduce resolution

2. Compute gaussian pyramid of images

1. For large motion (>1*pixel) use IR, CTFE

2. For violated spatial coherence use motion segmentation

3. For violated brightness constancy use exhaustive neighborhood search with normalized correlation

4. For temporal aliasing estimating wrong match use CTFE

1. Basic feature tracking framework

2. Track features by “following the arrows”

3. Real-time GPU implementations exists

1. Ambiguity of optical flow requires good features to track

2. Large motions requires discrete search

3. Changes in shape orientation and color requires matching flexibility

4. Due to dis- and occlusions, features have to be added and deleted

5. Drift yields errors maybe accumulating over time

1. Template is defined by small area around pixel

2. Match template against each pixel within search area

3. Match measure as SSD or correlation

4. After finding best discrete location use LKE for sub-pixel

1. Select features in first frame

2. For each frame

2.2 Update positions of tracked features

2.3 Terminate inconsistent tracks

2.4 Start new tracks if needed // ~10 frames

1. Find good features using eigenvalues

2. Use LKE and pure translation model

3. Check consistency of tracks by affine registration