Monocular Tracking 3D People with Back Constrained Scaled Gaussian Process Latent Variable ModelsJunbiao Pang1,2, Qingming Huang1,2, Shuqiang Jiang21Graduate School of Chinese Academy of Sciences, China2Institute of Computing Technology, Chinese Academy of Sciences, China{jbpang, qmhuang,sqjiang}@AbstractTracking 3D people from monocular video is often poorly constrained. To mitigate this problem, prior information can be exploited. In learning the prior stage, most algorithms think representing high-dimensional pose space in low-dimensional space as dimension reduction procedure, without considering the geometrical relation or time correlation in pose space. Therefore, the prior loses physical constrains in pose space. In this paper, the back constrained scaled Gaussian process latent variable model (back constrained SGPLVM), a novel dimensionality reduction method for learning human poses is proposed. The low-dimensional latent space is optimized for preserving geometrical relation or time correlation in high-dimensional pose space. The learned latent space is a smooth manifold. The smooth latent space will be satisfactory state space for tracking to avoid searching in high-dimensional pose directly. Experiment demonstrates that the back constrained SGPLVM integrated with particle filtering framework can track 3D people accurately and robustly, despite weak and noisy image measurements..Keywords: Particle filtering, Gaussian Process, Human motion analysis1IntroductionHuman motion analysis is important for many applications: video surveillance, gesture analysis, advanced human computer interface, content based video retrieval, etc. Tracking 3D people in monocular video is a fundamental problem for human motion analysis. However, this is a challenging problem because of poor constraints and high dimensionality of the state space. Features extracted form 2D images are subject to self-occlusions, ambiguities, and loose-clothing. These observations contain false detections, ambiguities and do not provided depth information. The estimation of the human poses involves searching in high-dimensional, complex, state space, thus, filtering based tracking algorithm can suffer the curse of dimensionality due to the high-dimensional human pose state space. These make tracking 3D people difficult. Prior information is essential to resolve the poor constraints. Se- Copyright © 2006, NLPR. This paper appeared at the Asia-Pacific Workshop on Visual Information Processing (VIP06), November 7-9, 2006, Beijing, China. Reproduction for academic, not-for profit purposes permitted provided this text is included.Figure 1.an overview of the tacking procedure. First, Obtaining the training data form motion capture data, and applying back constrained SGPLVM to model the latent space; second, given a video, tracking the 3D people with particle filtering in latent space; last, visualizing the predicted 3D pose. arching in the low-dimensional state space avoids curse of dimensionality.We propose to encode the sophisticated human poses with intrinsical low dimensional space. The learned prior mitigates the poor constraints, searching in lowdimen- sional space avoids the curse of dimensionality. Recently, SGPLVM in Grochow et al (2004) and Urtasun et al (2005), has been used to exploit the sophisticated physical constraints of human motion in low-dimensional space. Given training sequences, SGPLVM optimizes the low-dimensional latent space embedding of human pose space. The human poses have strong time correlation or geometrical relation between each other. However, the latent space in SGPLVM does not preserve these relations. Addressing this issue, a mechanism is needed to preserve these relations into low-dimensional space.In this paper, the back constrained SGPLVM, we propose, keeps geometrical relation or time correlation in latent space; thus, more physical constrains are keep into latent space as prior, these constrains make the latent space be a smooth and continuous manifold. During the tracking, the latent space will be satisfactory state space for tracking. Human pose propagates in latent space; the corresponding 3D human pose is projected into image plane for measurement. Experiment demonstrates tracking with back constrained SGPLVM prior is accurate and robust. Figure.1 describes the whole tracking procedure.The rest of the paper is organized as follows: Related work on 3D pose estimation will be discussed, in section 2. The back constrained scaled Gaussian process latent variable model is described in section 3. In section 4, we depict the tracking algorithm with learned prior. In section 5, experimental results are given. Conclusion is made in section 6.2Related workEstimating 3D pose is challenging because some degrees of freedom of the articulated model representing the human body are not observed. A large number of research papers address the topic of 3D human pose estimation. Paper, Gavrila (1999), provides reviews of earlier works in this area, but, in recent years, many other techniques have been proposed and we discuss them in the following.2.1Model-Based TrackingThese approaches presuppose an explicitly known parametric body model. Tracking the human body poses in image sequences earlier used gradient-based methods in Bregler et al (1998) and Demirdjian et al (2001), to track short sequences of human movement. Recently, the sampling-based algorithm has been adopted for human motion tracking by representing the pose estimated with a set of samples in Sminchisescu el al (2001), Deutscher el al (2000), Cham et al (1999), Sidenbladh et al (2000) and Sigal et al (2004). These statistical approaches can overcome some degree of pose ambiguities and could track for longer duration. However, due to the high dimensionality of the state space, the problem of particle degeneracy may still exist, as much of the state space is depleted of state samples. In the presence of significant pose ambiguity, these methods may also fail, unless other data (such as multi-views data) is available.2.2Learning-Based MethodThese approaches consist of estimating the 3D pose by matching features of an input image to those available in a large set of labeled image features containing all of the possible body poses. In these methods, different image features have been used including edges in Grauman et al (2003), silhouette outlines in Rosales et al (2004), and shape context descriptor in Agarwal et al (2004). In Agarwal et al (2004), a regression technique known as the relevance vector machine was proposed. These methods do not require a human body model and initialization. However, these methods require clear background subtraction and a large number of training data which is realistic captured or virtual synthesized from multi-views.2.3Priors-Based TrackingThe learned model as prior aids tracking because the prior consists of complex physical constrains of poses and mitigates the ambiguity.Recent works in Urtasun et al (2005), Li et al (2006) and Tian et al (2005), pay more attention on representation of high-dimensional human poses and motion. Urtasun et al (2005) utilizes scaled Gaussian process latent variable as prior model and a second order Markov model for tracking. Locally linear coordination (LLC) in Li et al (2006) is learned to solve the embedding of high-dimensional space, and the learned model is utilized in Multiple Hypothesis Tracker for tracking 3D people in low dimensional space. Tian et al (2005) employs the GPLVM and a modified CONDENSATION algorithm where samples are drawn from low-dimensional latent space for tracking 2D people. Tracking in low-dimensional space can avoid curse of dimensionality. However, the high-dimensional humanposes have strong time correlation. These above learnedpriors are only dimensional reduction procedure without considering the relation between poses, such as, time correlation or geometric relation.Our work is motivated, in part, by prior-based tracking. Geometrical relation in high dimensional space has beenpreserved in latent space. The back constrained SGPLVMhas advantage over other dimensionality reduction technique for keeping more geometrical constrains or time correlation of high-dimensional space.3Back Constrained Scaled Gaussian ProcessLatent Variable ModelBack constrained SGPLVM is a latent variable model,which preserves the geometric relation of poses space inlatent space. The back constrained SGPLVM is a versionof Gaussian process latent variable model (GPLVM) in Lawrrence (2004). GPLVM is the dual problem of probabilistic PCA (PPCA) in Tipping et al (1999).GPLVM models the prior knowledge as Gaussian Processin the form of kernel matrix, and marginalizes the parameters in PPCA as well as optimizes latent variablewith maximization likelihood methods3.1SGPLVMSGPLVM is GPLVM with scale for each dimension data.Due to different data dimensions have different intrinsicscales, a small change in some data affects other data dramatically. For instance, a small change in globalrotation of the human model affects the pose much morethan a small change in the wrist angle. We only describethe basic mechanism of SGPLVM here.Let1,...,[]TNY y y≡denotes a N D× matrix. ,1,...,iy i N=is one of the training poses from MoCap data withD dimension. Under the Gaussian process model, the conditional density distribution [12]:21222||1(|,)exp()2(2)||TDN DWp Y X S tr K YW YKπ−⎛⎞=−⎜⎟⎝⎠(1)Where1{,,,}S Wαβγ= is the unknown kernel parametersset.1[,..,]TNX x x=denotes a N q×matrix, ,1,...,ix i N= ispoints in latent space, with q dimension, usually q D.Everyix represents one of poseiy in low dimensionalspace.1(,...,)DW diag w w≡is the diagonal matrix containing scaling factor for each data dimension andK is the kernel matrix. The elements of kernel matrix aregiven by a kernel function, (,)ij i jK k x x=. Kernel matrixis the core of Gaussian process. The prior knowledge ismodeled in kernel matrix. The Radial Basis Function(RBF) kernel is usually used for nonlinear mapping.211,(,)exp(||||)2i j i j i jk x x x xγαδβ−=−−+ (2)Where,i jδ is the Kronecker delta function.Figure2. Visualization of latent space for walking cycle sequences. (a) shows 2D latent points modeled by SGPLVM. (b) shows 2D latent points modeled by back constrained SGPLVM. (c, d) show the 3D latent points modeled by back constrained SGPLVM form side and top views. (e, f) show 3D latent points modeled by SGPLVM form side and top views.The model simultaneously provides a smooth mapping from latent space to poses space and represents poses in latent space. 3.2 Back Constraints for Data RepresentationIn prior-based tracking, the prior knowledge should encode the relations between poses. We exploit the geometrical relation in poses data and represent the relation in latent space. In dimension reduction, it is common that preserving the geometrical relation between points in the original data. For instance, LLE (local linear embedding) in Roweis et al (2000), Isomap, and spectral embedding techniques in Belkin et al (2001) all have this characteristic. Lawrence (2006) proposes back constraints from high-dimensional space to low-dimensional space forkeeping geometrical relation. The basic idea of backconstraints is encoding relation in high-dimensional space into latent space. Back constrained function mapping :g y x →is introduced to adding geometrical relation into latent space.(;)nj j n x g y w = (3) Where n y is the pose data, nj x is the latent points.w is the function parameters. This function “passes” the relation between poses into latent space, for instance, time correlation or geometrical relation.For computation conveniently, we select the funct- ionn j g to be kernel based regression function.1(;)(,)Nj n jm n m m g y A a k y y ==∑ (4)Where 1,1{}jn j q n N A a ≤≤≤≤=is the back constraints matrix, and the kernel matrix is2(,)exp(||||)2n m n m k y y y y γ=−− (5)Where 2γis the inverse width parameter. We determine2γ empirically.The relation between back constraints matrix A and latent space X is determined by following equations.11[,...,][(,),..,(,)],1,...,n n nq Tn n N x x x k y y k y y A n N==⋅= (6)3.3 Learning the Model ParametersLearning back constrained GPLVM entails estimating the kernel parameters 1{,,}αβγand back constraintsmatrix {}ij A a =and scaled matrix {}ij W w =. Following Lawrence (2006), we maximize theposterior 1({},,,,{}|{})ij ij i p a w y αβγ, which corresponds to minimizing the following negative likelihood functionwith Scaled Conjugate Gradients. The initialized latent points can be obtained by probabilistic PCA. 221111ln ||||||ln222T k k k i N k ik k a D L K w Y K Y x w βγ−=+++∏∑∑ (7)The derivation of L with respect to parameters will bedescribed in appendix A.To speed up the training, the kernel matrix K is only learned on a subset of training data. This selected subset is called active set . The active set can be considered as sparse representation of training data. The process of selecting the active set is described in Lawrence (2003). The training algorithm will be described in appendix A as well. 3.4 Model Results Figures 2 shows results learned from motion capture data (from CMU MoCap database). The latent space is learned for walking cycle sequence. The human model wasparameterized with 10 selected joint angles with 459 training poses. In each case, before minimizing L , theglobal translation of human model is removed. Proba-bilistic PCA is used to obtain the initial latent points foriteration optimization. The hyperparameters are initially set to 1 with 32110γ−=×. The negative log posterior L was minimized using Scaled Conjugate Gradients.4 Monocular TrackingThe basic idea of tracking 3D people is that integrating the learned model into particle filtering framework. Let 1(,...,)t t Z z z ≡denotes the measurement when tracking. The complete body pose is controlled by a state vector (,)t t t s P x =. t P represents the global position of the body, and t x is the point in latent space. We manually initialize the 3D position t P and 3D pose 0y . Given 0y , the initial latent points 0x can be obtained by minimizing the following likelihood function in Grochow et al (2004) and Mackay (1998).222||(())||1(,)ln ()||||2()22i i i i i i i W y f x D L x y x x x σσ−=++ (8) Where 1()()T i i f x Y K x −= (9)21()(,)()()i i i i x k x x k x K k x σ−=− (10)K is the kernel matrix for active set, ()i k x is a vector inwhich the i-th entry contains (,)i k x x . During the optimization, 2||(())||W y f x −term tries to keep i y close tothe corresponding prediction ()f x , and 2ln ()x σterm tries to keep the i x value close to the training data. The21||||2x is a penalty term. During the tracking, the learned model provides prior of poses, the latent space is the state space instead of pose space. We model the state dynamics as first-order auto-regressive dynamics: 1,(0,)t t t t x Ax Cv v N +=+Σ∼ (11) We select the ,,A C Σempirically, A is 0.01, and C is 0.01, and Σis 0.01. The tracking algorithm can be described in particle filtering framework in Arulampalam (2000). The detail tracking algorithm will be described in figure 3.To compute the measurement for the current prediction and the input video frame, we use the same method proposed by Sigal (2004). First the silhouette of the current video frame is extracted through background subtraction, and the chamfer matching cost between the projected model and image silhouettes is considered to be proportional to the negative log-likelihood. 5 ExperimentsIn this section, we describe the experimental setting and the tracking results.5.1 Test Setting and Ground TruthThe proposed algorithm has been tested in tracking walking humans. The test set, calibration information and ground truth are obtained from Balan et al (2005). The testset has four i mage sequences, captured by four synchron-Figure3. The detail monocular tracking 3D people algorithmized cameras from different viewpoints. The image sequences show that a person walks cycle with variable speed. For tracking, the challenging of the test set is largemotion, motion blur, loose clothing and self-occlusions. The image sequences have been widely utilized as test set in Li et al (2006), Balan et al (2005) and Sigal et al (2004). We only use one of sequences for test, and calibrationinformation is utilized to project 3D model into imageplane. In the experiments, we maintain the 2 dimension latent pace and 200 particles for our algorithm. We use the same human model proposed by Sigal et al (2004), which consists of a group of cylinders. One training data form motion capture data has 1900 frames with 28 freedoms. The training data is used for learning prior by SGPLVMand back constrained SGPLVM, respectively. We compare our algorithm with annealed particle filtering in Deutscher et al (2003) because the annealed particle filtering has good performance in tracking 3D people. We set annealed particle filtering with 5 layers and 200 particles per layer as well as four synchronized views simultaneously for measurement.5.2 Tracking Results and DiscussionsWe compare above tracking algorithm on a 3.0 GHzmachine with 512 MB RAM. We develop the system on Matlab and do not optimize the code specially. SGPLVM and back constrained SGPLVM tracking algorithm take approximate 11 seconds per frame, because we sample thesame particle number. The annealed particle filtering takes approximate 40 seconds per frame. Figure 4 shows selected frames for comparing the trackingFigure.4. selected frames of tracking results with different prior. The 3D cylinder human model is projected into image plane. Upper row: tracking result with back constrained SGPLVM. Lower row: tracking result with SGPLVM.results. The upper row shows the tracking result with back constrained SGPLVM as prior, and the lower row depicts the result with SGPLVM as prior. At 102 frame in Figure.3, SGPLVM as prior underestimates left wrist position. At 104 frame in Figure.3, SGPLVM as prior underestimates the right wrist and right leg position. SGPLVM prior does not preserve the poses relation, or time correlation, and this default may cause above error in tracking. At 200 frame in Figure.3, SGPLVM as prior loses track of the left wrist. The broken trajectory in SGPLVM latent space may be its weakness. The large motion and continuous poses can not be represented by broken trajectory. In contrast, back constrained SGPLVM has continuous manifold in latent space which leads to temporal smoothness for tracking in latent space; hence, tracking with back constrained SGPLVM prior is more robust and reliable.A qualitative comparison will be carried out. Tracking with back constrained SGPLVM prior is compared with annealed particle filtering in Deutscher et al (2003). We compute the RMSE, as compute mean square root of sum of squared deviations from the ground truth for every frame. More specifically.RMSE=Where M is the number of freedoms, i j is the estimated i-th freedom,ij is the corresponding freedom in ground truth. Figure. 5 shows tracking results for two algorithms. As can be seen in the figure.4, tracking with back constrained SGPLVM can obtain the better results as annealed particle filtering, but, the annealed particle filtering uses 4 synchronized views for measurement. At some frames,our algorithm has better performance, such as, from 100Figure. 5 RMSE of the two algorithms.frame to 140 frame, from 180 frame to 200 frame. Elbows and curs has large motion, and self-occlusion and motion blur, therefore, we select right elbow joint and right knee joint to test accuracy of our algorithm. Figure.6 shows the accuracy of the representative joints. The maximum absolute deviation with ground truth is 0.38 degree in knee joint, 0.50 degree in elbow joint, respectively.Figure. 6 Comparing the tracking results with ground truth. Left: comparing right elbow joint; Right: comparing rightknee joint.Figure.6 some tracking results with back constrained SGPLVM prior.Visit: /en/project/mrhomepage/En_demo.htm#humanpose for video.Figure.6 illustrates some tracking result with back constrained SGPLVM prior. The tracking algorithm canreliably track more than 200 frames. In some frame, as thethird frame form left to right in lower row in Figure6, theprojection of 3D cylinder human model can not match thepeople perfectly. Global translation error may cause it.More different speed of training date may mitigate thissituation.6 ConclusionsWe have presented the back constrained SGPLVM to learn prior for tracking 3D people and have shown that the prior can be used effectively for monocular 3D tracking. The experiments demonstrate that smooth and continuous trajectory in latent space is suitable for tracking 3D people. It is shown that our algorithm is capable to handling self-occlusion, motion-blur. Further work will focus on dynamic updating the learned model and easier initialization for tracking.AcknowledgementThis work was supported by the NEC Research China and“Science 100 plan” of Chinese Academy of Sciences, as well as supported by Beijing Natural Science Fundation: 4063041. We thank A.Balan, L.Sigal and M.Black forproviding the motion capture data set and code forperformance comparison and thank N. Lawrence for providing the prototype code of GPLVM. Appendix The gradients of L can be computed with the help of thefollowing derivatives, along with the chain rule:111T T LK WYY W K dK K−−−∂=−∂ ()(,)i j i j iKx x k x x x γ∂=−−∂ 1,,[(,),...,(,)]Ti i i N i j i jx A k y y k y y a a ∂∂=⋅∂∂ 2exp(||||)2i j K x x γα∂=−−∂ ,i j Kδβ∂=∂211||||(,)2i j i j K x x k x x γ∂=−−∂ 22222()(())/()()||(())||[]/(2())()T TL f x W W y f x x x xx W y f x D x x xx σσσσ∂∂=−−+∂∂∂−−+∂ 1()()T f x k x Y K x x−∂∂=∂∂ Algorithm 2 summarizes the training steps as following:Algorithm An algorithm for training back constrained SGPLVMInitialise: latent points X through probabilistic PCA.A size for active set, d . A number of iterations, T For T iterations doSelect a new active set.Optimise (7) with respect to the parameters of K using scaled conjugate gradients. Select a new active set.For each point not in active set , j .doOptimise (8) with respect to ,i j a using scaledconjugated gradients.End EndReferencesGavrila, D.M. (1999): The visual analysis of human movement: a survey. Computer Vision and Image Understanding 73(1):2066-2073Sminchisescu, C. and Triggs, B (2001): Covariance scaled sampling for monocular 3D body tracking. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition . 447-454Deutscher, J. Blake, A. and Reid, I (2000): Articulated body motion capture by annealed particle filtering. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition . 126-133Cham, T. J and Rehg. J.M (1999): A multiple hypothesis approach to figure tracking. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition. 239-245 Sidenbladh, H, Black, M. J. and Fleet, D.J (2000): Stochastic tracking of 3D human figures using 2D image motion. In: Proc. European Conf. on Computer Vision. 702-718Tipping, M. E. and Bishop, C. M (1999): Probabilistic principal componet analysis. Journal of the Royal Statistical society : 611-622Lawrence, N.D, Seeger, N. and Herbrich, R. (2003): Fast sparse Gaussian process methods: The informativevector machine. In NIPSGrochow, K. Martion, S. L. Hertzmannn, A. and Propovic, Z. (2004): Style-based inverse kinematics. In: ACM Computer Graphics(SIGGRAPH): 522-531 Balan, A. Sigal, L. and Black, M.(2005): A Quantitative evaluation of video-based 3D person tracking. IEEE workshop on VS_PETS , 349-356Urtasun, R., Fleet, D.J., Hertzmann and A. Fua, P. (2005): Priors for people tracking from small tracking sets. In: Proc. IEEE International Conf. on Computer Vision . 403-410Li, R. Yang, M. H. Sclaroff, S. and Tian, T.P (2006) Monocular tracking of 3D human motion with coordinated mixture of factor analyzers, In: Proc. European Conf. on Computer VisionLawrrence, N.(2004): Gaussian process latent variable models for visualization of high dimensional data. In NIPSMackay D (1998): Introduction to Gaussian processes. In C. Bishop, editor, Neural Networks and Machine Learning NATO ASI Series, 133-166Lawrence, N. Candela. J.Q (2006): Local distance preservation in the GP-LVM through back constraints. In: International Conf. on Machine LearningArulampalam, S., Maskell, S., Gordon, N., Clapp, T (2000): A tutorial on particle filters for on-line non-linear/non-Gaussian Bayesian tracking.Sigal, L., Bhatia, S., Roth, S, Black, M. J, Isard,M (2004): Tracking loose-limbed people. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition : 421-428 Tian, T, Li. R., and Sclaroff, S. Articulated pose estimation in a learned smooth space of feasible solutions. CVPR Learning workshop .:2005Roweis, S. and Saul, L.(2000): Nonlinear dimensionality reduction by locally linear embedding. Science ,290:2323-2326Belkin, M. and Niyogi, P.(2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. NIPS 585-591Bregler, C and Malik, J (1998): Tracking People with twists and exponential maps. CVPR, 8-15Demirdjian, Q. and Faugeras, O (2001): 3D Articulated models and multi-view tracking with physical forses. CVIU , (81),328-357Grauman, K. Shaknarovich, G. and Darrell, T (2003): Inferring 3D structure with a statistical image-based shape model. ICCV ,641-648Rosales, R. Siddiqui, M. Alon, J and Sclaroff, S (2001) Estimating 3D body pose using uncalibrated cameras. CVPR , 821-827Agarwal, A. and Triggs, B (2004): 3D human pose from silhouettes by relevance vector regression. CVPR , 882-888。
