Stereo Correspondence Using Segment Connectivity

stere orectif y原理1.引言在计算机视觉领域中，立体视觉是一个重要的研究方向。

通过利用多个摄像头或相机来获取不同视角的图像，我们可以实现诸如深度感知、三维重建和目标检测等应用。

而立体矫正（s t er eo re ct if ic ati o n）是立体视觉中的一个关键步骤，用于对立体图像进行预处理，以便后续的像素匹配和深度计算更加准确。

2.立体视觉基础在了解立体矫正原理之前，我们需要了解一些立体视觉的基础知识。

2.1本质矩阵本质矩阵（e ss en tia l ma tr ix）是描述两个相机之间几何关系的矩阵。

它可以通过摄像头标定得到，是一个3x3的矩阵。

本质矩阵中保存了两个视角之间的相对位置和姿态信息。

2.2极线约束极线约束（e pi po lar c on st ra in t）是立体视觉中一个重要的性质。

假设在一个图像中找到了一个特征点，那么它在另一个图像上的对应点必定在相应的极线上。

这是由于在立体几何中，两个相机的投影光线都通过空间中的一个特定点，从而形成了极线。

3.s t e r e o r e c t i fy原理s t er eo re ct if y的目的是将不同视角下的立体图像进行矫正，使得它们在同一平面上，并且满足极线约束。

这样可以简化后续的图像处理过程。

s t er eo re ct if y原理如下：3.1计算本质矩阵首先，我们需要通过摄像头的标定参数计算出本质矩阵。

摄像头标定是确定相机内参和畸变参数的过程，通过已知的标定板图像来进行。

利用标定参数，我们可以得到两个视角之间的本质矩阵。

3.2计算r e c t i f i c a t i o n变换接下来，我们需要计算用于矫正图像的re c ti fi ca ti on变换。

r e ct if ic at io n变换是一种刚性变换，它将两个视角的相机坐标系变换到同一平面上，并且使得两个相机光轴平行。

3.3应用r e c t i f i c a t i o n变换将r ec ti fi ca ti on变换应用到图像上，即可实现立体图像的矫正。

Decisions require actions to be performed on devices Decisions are frequently not elementary device interactions but rather relatively complex commands
x y
t t

Example: A differential drive robot
) ) r ( r ( L R L R v x cos( ) , v y sin( ) 2 2 r L R d

First industrial robot


1969: Stanford Arm

Autonomous mobile research robot Dextrous, electric motor driven robot arm
Unimate
Robots

Robot Manipulators

Autonomy


Intuitive Human-Robot Interfaces

Robots have to be capable of achieving task objectives without human input Robots have to be able to make and execute their own decisions based on sensor information Use of robots in smart homes can not require extensive user training Commands to robots should be natural for inhabitants

Quasi-Dense Wide Baseline Matching Using Match PropagationJuho Kannala and Sami S.BrandtMachine Vision GroupUniversity of Oulu,Finland{jkannala,sbrandt}@ee.oulu.fiAbstractIn this paper we propose extensions to the match propa-gation algorithm which is a technique for computing quasi-dense point correspondences between two views.The ex-tensions make the match propagation applicable for wide baseline matching,i.e.,for cases where the camera pose can vary a lot between the views.Ourfirst extension is to use a local affine model for the geometric transformation between the images.The estimate of the local transformation is ob-tained from affine covariant interest regions which are used as seed matches.The second extension is to use the second order intensity moments to adapt the current estimate of the local affine transformation during the propagation.This al-lows a single seed match to propagate into regions where the local transformation between the views differs from the initial one.The experiments with real data show that the proposed techniques improve both the quality and coverage of the quasi-dense disparity map.1.IntroductionAutomatic three-dimensional model acquisition for an unknown scene from multiple images is a classical prob-lem in computer vision.For uncalibrated image sequences, the standard approach is based on sparse points of interest which are tracked across the sequence and reconstructed to-gether with the camera motion[5,3,6].Eventhough the sparse approach is often sufficient for calibrating both the internal and external parameters of the cameras it is not suf-ficient for the accurate three-dimensional description of the scene.However,the sparse approach is often used to sim-plify the problem whereafter dense matching methods are applied for the calibrated views[16,17].So far there are many dense matching algorithms avail-able.The number of traditional two-frame stereo corre-spondence algorithms is large[18];there are also several volumetric approaches which are voxel-oriented[19,8]or based on graph cuts[7]or level sets[2].In addition,prob-abilistic[20]and PDE-based formulations[21]have beenused for dense matching of wide baseline images.The prob-lem with traditional stereo algorithms is that they are de-signed for view pairs with a small baseline and for almost frontoparallel planes.Hence,they can not be easily used for wide baseline views for which the epipolar lines are not parallel.On the other hand,the volumetric approaches use a discretized volume or restrict the possible depth values to a predefined accuracy.Thus,these methods are expensive in terms of time and memory when high accuracy is needed.However,the quasi-dense approach[10]is more efficient than the dense approaches but sufficient also for3D surface reconstruction.This paper is hence built on the previous works[9,10] where a quasi-dense approach to surface reconstruction from image sequences was proposed.In this approach, sparse interest point matches are used as seed points for the match propagation algorithm which produces denser,but not completely dense,pixel correspondences that are called quasi-dense pixel correspondences.The quasi-dense pixel correspondences are computed for successive image pairs in the sequence and they are also used in the image geometry estimation.For uncalibrated image sequences,the quasi-dense approach provides a full automatic geometry estima-tion like the standard sparse approach but it also providesa high density of3D points on which a surface representa-tion can be reconstructed.Moreover,it has been reported that the quasi-dense approach provides more robust and ac-curate geometry estimation than the sparse approach[10].Hence,even if thefinal goal would be a completely dense reconstruction the quasi-dence approach may still be use-ful since most of the dense methods,such as[21],require reliable pixel correspondences for initialization.There are certain issues in the match propagation algo-rithm which limit its applicability in the wide baseline case.At each step of propagation,small image patches are ex-tracted around the current seed point in both images and the new candidate matches are scored according to the zero-mean normalized cross-correlation(ZNCC)[9].Hence,it is implicitly assumed that the local transformation between the patches is effectively a translation,so the algorithm is1not suitable for wide baseline matching.To cope with this problem,we use a general affine model for the local trans-formation between the patches.Moreover,we will show that it is possible to determine the local affine transforma-tion adaptively during the propagation.This is achieved by using the second order intensity moments locally together with the epipolar geometry.The experiments show that the proposed extensions improve the performance of match propagation in the wide baseline setting.The structure of this paper is as follows.First,there is a brief description of the original match propagation algo-rithm in Section2,and the proposed improvements are then described in Sections3and4.The experiments are pre-sented in Section5.2.BackgroundThe match propagation algorithm[9]starts from a set of sparse matches between two images,I and I ,and produces a quasi-dense disparity map which contains a large num-ber of pixel correspondences.In[9],the initial matching of sparse interest points was done by using the ZNCC score to-gether with a simple cross-consistency check.The propaga-tion itself starts from the set of initial matches{(x i,xi )}i,where x i and xi are corresponding pixels in the two images.The initial matches are called seed points.The propagation proceeds by iterating the following three steps:(i)the seed point(x i,xi )with the best ZNCC score isremoved from the list of seed points(ii)new candidate matches are searched from the spatial neighborhood of the current seed(x i,xi)(iii)the candidate matches exceeding a ZNCC threshold and a difference-based confidence limit are stored in the disparity map and added to the list of seed points In this way the number of pixel correspondences in the dis-parity map increases until the list of seeds becomes empty. More precisely,in step(ii)above the spatial neighbor-hood,N(x,x ),of a match(x,x )is defined so that the matches in this neighborhood satisfy a constraint called dis-crete2D disparity gradient limit.The precise definitions are as follows.The(2N+1)×(2N+1)neighborhoods of pixels x and x areN(x)={u|(u−x)∈[−N,N]2},N(x )={u |(u −x )∈[−N,N]2},as usual,and the neighboring matches satisfying the dispar-ity gradient constraint are given byN(x,x )={(u,u )|u∈N(x),u ∈N(x ),||(u −x )−(u−x)||∞≤ },where is the disparity gradient limit.With integer pixel coordinates,the smallest nonzero value for is1which is also the value used in[9](together with N=2).Enforcing the disparity gradient constraint implies that the2D vectors from the current seed point to the new candidate point must have approximately the same direction in both images.In step(iii)of the algorithm,when the candidate matches (u l,u l)are given for the current seed,a difference-based confidence measure s is computed for the candidate pixels, s(u l)=max{|I(u l+δ)−I(u l)|,δ∈{(±1,0),(0,±1)}}, and s (ul)is computed in the corresponding way.In order to prevent propagation into too uniform areas,those candi-date matches are discarded for which s(u l)<t or s (ul)< t,where t is a predefined threshold.The remaining candi-date matches are sorted using the ZNCC measure,which is computed using the windows of size(2W+1)×(2W+1) around the candidate pixels.Those matches which are not yet in the disparity map(either one of the matching pixels) and exceed a predefined threshold z for the ZNCC score are stored in the disparity map and added to the list of seeds. The match propagation algorithm is efficient.When the ordering of seed points according to the ZNCC scores is maintained in a heap data structure the time complexity of the algorithm is O(n log n)where n is thefinal number of matched pixels.The algorithm is also robust to outliers ly-ing in the set of initial matches due to the best-first strat-egy in the propagation[9].In addition,if the fundamental matrix for the view pair is known,the epipolar constraint can be used as an additional matching constraint during the propagation.In[10],in fact,the unconstrained propaga-tion isfirst used to generate quasi-dense point correspon-dences for the fundamental matrix estimation and there-after the constrained propagation is used to produce thefinal matches.The major problem with the match propagation algo-rithm is that it requires conventional stereo image pairs.In particular,the orientation of the images should be similar due to the usage of the direct ZNCC similarity measure and the aforementioned disparity gradient limit.However,in the following we will show that using an affine model for the local transformation between the images makes the match propagation approach applicable in the wide-baseline case.3.Match propagation for wide baseline views Thefirst step towards quasi-dense wide baseline match-ing is to use the sparse wide baseline matching approaches [15,14]for initial matching.Hence,we suggest using affine covariant regions[15]as seed matches for the match prop-agation.Such seed matches can provide an estimate for the local geometric transformation between the images,in ad-dition to the position of the match.In order to properly uti-lize this information,we propose an extension to the matchpropagation algorithm which allows adequate modeling of geometric transformations between image patches.But first,we briefly describe the computation of seed matches.3.1.Initial matchesThe affine covariant region detectors[15]together with a viewpoint invariant descriptor[12]have shown good per-formance in sparse wide baseline matching.Hence,it is a natural choice to use this approach in computing the ini-tial matches for match propagation.In fact,the idea of propagating matches from few initially matched regions to nearby ones has been used in the context of object recogni-tion[4].The detector that we used in our experiments was the Hessian-Affine detector proposed in[13].The detected regions were matched using the SIFT descriptor[12].In principle,any other region detector could be used as well, as long as the detected regions can be represented by el-lipses so that the ellipses of corresponding regions,given as symmetric2×2matrices C and C ,are related byC =A− CA−1,(1) where A is the local affine transformation between the matching regions[13].Equation(1)provides a basis for estimating the affine transformation A given the measure-ment regions C and C .In fact,from(1)we getA=C −1/2RC1/2,(2) where R is an arbitrary orthogonal matrix.Additional in-formation is needed to determine R and here we used the orientation of local image gradients as in[12,13].In sum-mary,the result of the initial matching step is a set of pointmatches{(x i,xi )}i(the centroids of the matched ellipticalregions)accompanied with the estimates A i for the corre-sponding local transformations.Giving an example,Figs.1(a)and1(b)illustrate the matched Hessian-Affine regions in two images of a planar scene.The images are from[22]where also the known ho-mographies between the views are available together with the implementations of region detectors[15]and descriptors [14].For the descriptor matching we used a modified near-est neighbor strategy.We required both that the matching descriptors are the nearest neighbors of each other and the distance to the second nearest neighbor is above a certain threshold.In our experiments,we found that such matching strategy significantly reduces false matches for images with repeating texture patterns as opposed to using just a single threshold for the nearest neighbor distance.In Fig.1,the50 best matching regions are shown.3.2.Propagation with affine normalizationIn this section we describe ourfirst extension to the match propagation algorithm.We append the seed matches(a)(b)10020030040050060070080010020030040050060012345671002003004005006007008001002003004005006001234567them,denoted by the cross,is used as a seed match for the match propagation algorithm.The pixel matches computed with the orig-inal(c)and the modified(d)propagation approach are illustrated in the second view by coloring them according to their Sampson distance from the ground truth homography.(The distances over5 are suppressed to5,the grayvalue for the noncommon area is6.)with an affine transformation matrix which is used to nor-malize the local image patches before enforcing the2D dis-parity gradient limit and computing the ZNCC scores.In addition,we use a subpixel level of accuracy for the po-sition of seed points whereas the disparity map is stored at the pixel level resolution.Although the proposed im-provements seem to be minor,they significantly improve the matching result when carefully implemented.As before,the initial seed matches,consisting of points {(x i,x i)}i and matrices A i,arefirst scored using the ZNCC measure.The ZNCC scores are computed from geo-metrically normalized image patches where the normaliza-tion is done as follows:(a)a(2W+1)×(2W+1)square patch centred on the seed point is extracted from the image which locally has a lower resolution(the local magnifica-tion factor is given by|det A i|),(b)the corresponding area in the other image is determined by affinely mapping the square patch on the other image,(c)the patches from both images are transformed to(4W+1)×(4W+1)windows by interpolation,(d)the interpolated windows are decimated to the size(2W+1)×(2W+1).This process is illustrated in Fig.2and,for a correct match,it should ideally result in two identical image patches when the affine transformation model is tenable.The scoring of seeds is not the only place where geomet-ric normalization is needed during the propagation.At each iteration,the whole neighborhood from which new matches are searched needs to be normalized.Therefore,in order to reduce the number of interpolations per iteration,we ac-tually compute the normalized patcheswindow of size (2(W +N )+1)×(2(W +N )lower resolution image.The normalized illustrated in Fig.3and they can be treated as in algorithm,i.e.,the disparity gradient limit and measure can be well used after the the pixels of the normalized patch do not ger pixel coordinates in the higher resolution we use subpixel level of accuracy for the seed matches.The coordinates are rounded to integer pixel values only when used to index the disparity map,for example when storing new matches.Importantly,when new matches are added to the list of seeds they inherit the affine transformation matrix from the current seed.This implies that a seed match always contains an estimate of the local affine transformation and it is the basis for the geometric normalization.In the normalization process,as shown in Fig.2,the im-age patch of the lower resolution image is first upsampled and then downsampled.This is necessary in order to cope with small scale changes between the images,i.e.,the up-sampling is needed to prevent aliasing effects in the interpo-lation of the higher resolution image and both image patches have to be processed in a similar way to assure comparabil-ity.In addition,the decimation of the transformed patches can be performed efficiently with two one-dimensional anti-aliasing filters.In fact,despite the inevitable interpolations,the extended match propagation algorithm is efficient.It is clear that the overall complexity remains the same and de-pends only on the number of final matches.In fact,we save some time by skipping the check of the confidence measure limit described in Section 2.We additionally found that the geometric normalization allows using higher threshold for the ZNCC score and this threshold alone is often sufficient for preventing the propagation into too uniform areas.The necessity for the affine normalization is made clear in Fig.1,where the poor performance of the original prop-agation approach is illustrated with the disparity map in Fig.1(c).In this example the propagation was started from a single seed match,denoted by the cross in Figs.1(a)and 1(b),using the default parameter values (N =2, =1,t =0.01,W =2,z =0.5)[9].The number of pixel cor-respondences in the final disparity map is 10726but most of them are not correct as their Sampson distance [6]to the ground truth homography is large.On the contrary,the propagation with affine normalization performs well as shown Fig.1(d).There are 261572matches and the param-eter values used were N =2, =1,W =2and z =0.8.De-spite the higher correlation threshold the number of matches is much larger than in Fig.1(c)and most of them are correct.The computation time per match was 1.7ms for Fig.1(c)and 3.5ms for Fig.1(d)when Matlab implementations of both algorithms were used.4W +12W +1decimationFigure 2.The geometric normalization of local image patches.ˆxˆuˆxˆuFigure 3.The normalized image neighborhoods of size (2(W +N )+1)×(2(W +N )+1)for a seed point (x ,x ).Here N =2,W =1,and ˆ·refers to the symbols in the normalized frame.The ZNCC score for the candidate match (u ,u )is computed using the black framed windows.4.Adaptive propagationIn this section we propose our second extension to the match propagation algorithm which allows the adaptation of the local affine transformation estimates during the prop-agation.The adaptation is based on the second order in-tensity moments which determine the local affine transfor-mation up to a rotation.The remaining rotational degree of freedom is determined from the epipolar geometry which is assumed to be known.The motivation for proposing an adaptive propagation technique is to allow propagation into regions where the lo-cal transformation between the views differs from the ini-tial one.In fact,in the affine normalization scheme,pro-posed in Section 3,the estimate of local affine transforma-tion is inherited from the current seed to its descendants.This implies that each initial seed match can propagate only into such regions where the estimated transformation,orig-inally computed from the affine covariant interest regions,is valid.In some cases this may be sufficient,for example,when the global transformation between the views is close to affine,as in Fig.1,or when the initial seed regions are dense enough so that the local transformations in different parts of the scene can be modelled.However,in some cases a more powerful propagation strategy would be useful.In Fig.5we have two images of a scene containing two planes,a map on a table and a calibration object.Here the affine transformation recovered from one region match,located in the front and denoted by yellow ellipses,does not ade-quately model the transformation of the whole plane since the perspective distortion is significant.In order to address this kind of situations,we propose the principle of adaptive propagation in the following.4.1.Local shape from intensity momentsLet f,f :R2→[0,1]denote the image intensity func-tions so that f (x)=f(A−1x)where A is a nonsin-gular2×2affine transformation matrix.Here we have dropped the translational part as we assume that the coor-dinate systems are centred to the points under considera-tion.Given two positive window functions g and g so that g (x)=g(A−1x)/|det A|,we define the windowed sec-ond moment matrix of f byS f,g(x)= vv f(v)g(x−v)d v,(3)and the matrix S f ,g of f is defined in a similar way.Since the affine transformation between the window functions is the same as between the intensity functions we getS f ,g (x)=AS f,g(A−1x)A .(4) By introducing a simpler notation at the origin,S = S f ,g (0)and S=S f,g(0),we get from(4)thatA=S 1/2RS−1/2,(5)where R is an orthogonal matrix which can be determined from a pair of corresponding directions in the two images (i.e.from one unit vector correspondence pair).Hence,the equation(5)implies that the moment matrices could be used to determine the local affine transformation in a similar way as the elliptical interest regions were used in Section3.1. In general,the problem of using(5)is that if g isfixed and A is unknown,g is also unknown.However,in our application we usually have a relatively good guess for the local transformation in the neighborhood of the initial matches.Hence,at each propagation step,we use the affine estimate of the current seed in forming the window func-tions and thereafter the estimate is updated for the new seeds using(5).In detail,let us consider one step of the adaptive prop-agation process with the illustrations in Figs.2and3.As-sume that the current seed is(x,x )and the corresponding transformation estimate is A0.The normalization process is carried out as described in Section3.2and the resulting nor-malized image patches are shown in Fig.3.Now,consider that the normalized cross-correlation of the black framed subwindows in Fig.3exceeds the threshold so that(ˆu,ˆu ),corresponding to a match(u,u )in the original images,is a new seed match.Instead of performing the integration(3)in the original images we integrate in the normalized domain and thereafter transform the obtained moment matrices to the image coordinate frames.If the characteristic function of a(2W+1)×(2W+1)window,centred onˆu andˆu ,is used as a window function in the normalized patches the integration in(3)reduces to a simple summation.Then the obtained moment matricesˆS andˆS are transformed back to the original frames by S=ˆS and S =A0ˆS A 0. The new transformation estimate for the new seed match is then obtained by using(5).Instead of uniform weighting of the square window by its characteristic function a different weight function can also be used.For example,a gaussian window function of size(2W+1)×(2W+1)was found to give good results.The advantage of performing the in-tegration in the normalized domain is that the values of the windowing function can be computed in advance since the domain of integration is always the same.This significantly improves the efficiency.The idea behind our approach,presented above,is sim-ilar in spirit to those presented in[11]and[13].However, here the context of application and the implementation are different.Importantly,the shape adaptation is not based on the windowed second moment matrix of the intensity gra-dient as in[11,13]but on the windowed second moment matrix of the intensity function itself.In our experiments we found that,instead of the gradient,the intensity func-tion provides a more stable basis for estimating the local affine transformation between a pair of views.This is also intuitively reasonable since the gradient is more sensitive to noise.In addition,computing the windowed gradients dur-ing the propagation would increase the computational load of the quasi-dense approach.4.2.Photometric normalizationThere is a one important issue related to(3)that should be taken in to account in a careful implementation of the al-gorithm.If there additionally is an affine photometric trans-formation between the images,f (x)=λf(A−1x)+b,the transformation rule(4)does not hold.This is a drawback since the ZNCC measure is invariant to affine photometric transformations and,hence,the adaptive extension might be a bottleneck in applying the algorithm in varying illumina-tion conditions.Therefore,instead of applying(4)directly to the intensity functions wefirst remove the effect of pho-tometric transformations by the following local normaliza-tions˜f(x)=f(x)−µ(f)σ(f)+C,(6)˜f (x)=f (x)−µ (f )σ (f )+C,(7)Figure lines.The two-fold ambiquity is solved using the jointly oriented epipoles [1].where C is a positive constant,the mean µ(f )=f (x )g (x )d x / g (x )d x ,the standard deviation σ(f )=( (f (x )−µ(f ))2g (x )d x / g (x )d x )1/2.The correspond-ing quantities for f are defined in a similar way using the window function g .Here the constant C is added in order to preserve the positive (semi)definiteness of matrices S ˜f,g and S ˜f ,g .The value C =2was used in all of our experi-ments.After the normalization we proceed as above usingthe normalized intensity functions ˜fand ˜f instead of the original ones.4.3.Local orientation from the epipolar constraintAs described above,the windowed second order inten-sity moment matrices allow to recover the local affine trans-formation up to a rotation.The unknown rotation in (5)can be determined from one pair of corresponding unit vectors [13].Such corresponding unit vectors can be determined from corresponding epipolar lines using the concept of joint orientation of epipoles which determines the mutual posi-tion of cameras [1].The details are as follows.Consider the two view setting illustrated in Fig.4.There a 3D point X on a smooth surface is imaged with two pin-hole cameras placed at O and O .The tangent plane of the surface at X is Γand its intersection with the epipolar plane,defined by points X ,O and O ,is the line ˜l .Hence,under the assumption of locally planar surfaces,the epipolar lines l and l are images of the same 3D line in the neighborhoods of the corresponding image points x and x .Hence,the epipolar lines give a line correspondence from which cor-responding directions may be determined up to sign,i.e.,the corresponding direction for d in Fig.4is either +d or −d .The correct choice is of course such that the direc-tions in both images correspond to the same direction ˜don the scene plane.Given the fundamental matrix alone one may now compute the oriented epipoles e and e so that the corresponding epipolar lines l =e ×x and l =x ×e have proper joint orientation,i.e.,the vectors d =(l 2,−l 1) andd =(l 2,−l 1)indicate corresponding directions in the images.Here we assume that the image coordinate bases of both cameras have equal handedness [1].5.ExperimentsWe experimented the proposed extensions to the match propagation algorithm with real images.In the first experi-ment (Section 5.1)we had two views of a scene containing planar surfaces.In the second experiment we used three views of a complex 3D scene as described in Section 5.2.5.1.A scene with planar surfacesIn Fig.5there are two images of a scene which con-tains two planes,a map on a table and a calibration plane orthogonal to the plane of the map.This image pair is suitable for evaluating the algorithms since the calibration plane allows to compute the homographies which describe the mappings between the planes;using the homographies the found matches can be verified.In addition,it can be seen that there is a clear projective distortion between the views so that affine mapping is not a good approximation for the homographies.In order to assure that the recovered homo-graphies are accurate we calibrated the intrinsic parameters of the camera beforehand and removed the radial distortion from the images.The results of match propagation,started from a single seed match,using both the original approach and the pro-posed modifications are illustrated in Fig.5.The results in-dicate that the affine normalization improves the matching in the neighborhood of the initial match as far as the affine transformation estimate is valid.Utilizing the epipolar con-straint reduces the number of false matches,as expected.The best results are achieved using the adaptive propagation approach.The results with two different window functions,uniform and gaussian,are shown and it can be seen that the gaussian window performs slightly better.The largest difference between the last two disparity maps is on rela-tively uniform areas where the matching is more unreliable.In fact,if the reliability of matches was critical,one could prevent propagation into too uniform areas by using an ad-ditional threshold for the variance of the correlation win-dows.The coverage of the last two disparity maps in Fig.5is extensive verifying that the adaptive propagation works in practice.The matching is not completely dense,and there are quite many unmatched pixels in the upper part of the image.However,this is mostly due to the different lo-cal magnification factors in the images,especially in the back regions of the ground plane.The improvement,illus-trated in Fig.5,is achieved without significantly reducing the efficiency of the approach.The computation times per match for the subfigures (c)-(h)were 7.2,5.9,3.8,3.3,4.9and 3.6ms ,respectively,using our current Matlab imple-mentations.Here the relatively long computation times of the original approach are probably due to the fact that the ZNCC measure is computed many times without obtaining。

Depth Estimation and Image Restoration Using Defocused Stereo PairsA.N.Rajagopalan,S.Chaudhuri,Senior Member,IEEE,andUma MudenagudiAbstract—We propose a method for estimating depth from images captured with a real aperture camera by fusing defocus and stereo cues.The idea is to use stereo-based constraints in conjunction with defocusing to obtain improved estimates of depth over those of stereo or defocus alone.The depth map as well as the original image of the scene are modeled as Markov random fields with a smoothness prior, and their estimates are obtained by minimizing a suitable energy function using simulated annealing.The main advantage of the proposed method,despite being computationally less efficient than the standard stereo or DFD method,is simultaneous recovery of depth as well as space-variant restoration of the original focused image of the scene.Index Terms—Defocus,stereo,disparity,Markov random field,blur identification, depth recovery.æ1I NTRODUCTIONA N important area of research in computer vision is the recovery of 3D information of a scene from2D images.In humans, stereoscopically presented images provide information about depth.Julesz[1]showed that random dot stereograms provide a cue for disparity even when each image does not provide any high-level cue for depth.Interestingly,Pentland[2]reported a finding that the gradient of focus inherent in biological systems is also a useful source of depth information.Binocular stereo matching is,in general,ambiguous if the matching is evaluated independently at each point purely by using image properties.All stereo matching algorithms examine candidate matches by calculating how much support they receive from their local neighborhood.Marr and Poggio[3]proposed a cooperative stereo algorithm based on a multiresolution framework.Barnard and Thompson[4]proposed a feature-based iterative algorithm to solve the correspondence problem.A large number of papers have appeared in the literature on stereo analysis and a review of them can be found in[5].Conventional stereo analysis assumes an ideal pin-hole camera model which offers an infinite depth of field. However,any practical camera system will produce depth-related blurring.In the depth from defocus(DFD)technique,two images of an object,which may or may not be focused,and acquired with different camera settings,are processed to determine depth.The relative blur between the defocused images serves as a cue for depth. Unlike stereo,DFD uses a real aperture camera model(which is more practical).Since the early works of Pentland[2],several approaches [6],[7],[8],[9],[10]have emerged for solving the DFD problem.A comparative analysis of DFD and stereo can be found in[11].Related works on defocus blur estimation in conjunction with image motion/ disparity are addressed in[12],[13],[14],[15].It is well-known that stereo yields accurate estimates of depth but the depth map is sparse since correspondence can be obtained with confidence only at prominent feature points.On the other hand,DFD gives a dense depth map but the accuracy of depth estimates are generally inferior to that of stereo-based methods.Estimates of depth from defocus can be particularly poor in severely blurred regions[8]. In this paper,we relax the assumption of a pin-hole camera model and propose an algorithm to recover depth from defocused stereo (DFDS)pairs of images.Defocus and stereo cues are fused to obtain improved dense estimates of depth.The additional constraints provided by stereo help in refining the estimates of depth over those obtained using DFD alone.Tsai et al.[16]have also proposed a scheme for integrating stereo and defocus.But,they use estimates from DFD to only initialize their stereo matching algorithm.In stereo,the disparity is directly related to depth,while in DFD, it is the blur parameter that relates to depth.Hence,disparity can be expressed in terms of the blur parameter,the lens settings,and the base-line distance.This information is used to fuse the two methods, thereby deriving the advantages of both the methods.In DFD,since the blur depends on the depth of the scene,the point spread function (PSF)in turn becomes a function of depth.For simplicity,most techniques assume local space-invariance and compute depth. However,this can lead to poor estimates of depth due to the image overlap problem[9].We model the depth and the focused image of the scene individually as Markov random fields(MRFs).Our approach avoids windowing and addresses the DFDS problem in its generality.Given two defocused stereo pairs of images,we obtain a focused image of the scene and a dense depth(blur/disparity)map. An important advantage of our method is that it not only recovers estimates of depth but also performs effective space-variant image restoration.The performance of the method is validated on synthetic as well as real images.The accuracy of depth estimates is superior compared to those obtained from DFD or the stereo method.A dense depth map is estimated without correspondence and interpolation.The quality of the restored image is also quite good.Section2describes the framework for fusing defocus and stereo.In Section3,we discuss the proposed approach for solving the DFDS problem.Experimental results are given in Section4 while Section5concludes the paper.2F USION OF D EFOCUS AND S TEREOThe basic structure of our scheme is given in Fig.1.Note that we have two stereo pairs of images.The blurring of the two stereo pairs are different and so is their disparity since the image pairs are captured with two different camera settings.We attempt to simultaneously estimate blur(disparity)and restore one of the focused images of the scene in the defocused stereo pairs(say the left image).Estimating the other stereo pair is trivial once we know the disparity.As in most literature on stereo,we assume epipolar line constraint so that the disparity is only along one direction,say the j-direction.For the given observation model,the right image is given byx Rði;jÞ¼x Lði;jþdði;jÞÞþwði;jÞ;ð1Þwhere dði;jÞis the disparity associated with the stereo pair at locationði;jÞ.The noise wði;jÞis assumed to be zero-mean and white Gaussian.Because we use a practical real aperture camera model,points not in focus will appear blurred and the blur parameter k for the k th lens setting is given byk¼ r k V k1kÀ1kÀ1;k¼1;2;ð2Þwhere is a camera constant,r k is the radius of the lens aperture, F k is the focal length,and V k is the image plane-to-lens distance for the chosen lens-setting[8].Given two defocused images of a scene captured with different sets of camera parameters,the blur parameter at locationði;jÞfor the two defocused images can be shown to be related by1ði;jÞ¼ 2ði;jÞþ ;ð3Þ. A.N.Rajagopalan is with the Department of Electrical Engineering,Indian Institute of Technology,Chennai600036,India.E-mail:raju@ee.iitm.ernet.in..S.Chaudhuri is with the Department of Electrical Engineering,Indian Institute of Technology,Mumbai400076,India.E-mail:sc@ee.iitb.ac.in. .U.Mudenagudi is with the Department of Electrical Engineering,B.V.B College of Engineering,Hubli580031,India.E-mail:umakm@. Manuscript received4Sept.2002;revised29Oct.2003;accepted8Apr.2004. Recommended for acceptance by R.Kumar.For information on obtaining reprints of this article,please send e-mail to: tpami@,and reference IEEECS Log Number117241.0162-8828/04/$20.00ß2004IEEE Published by the IEEE Computer Societywhere¼r1V1r2V2and ¼ r1V11F1À1V1À1F2þ1V2:Thus, and are known constants that depend on the camera settings.Most DFD methods assume a Gaussian-shaped blur modelfor the camera PSF,i.e.,hði;jÞ¼12 2expðÀi2þj22 2Þ.Though the Gaussianblur is of infinite extent,a finite spatial extent approximation (Æ3 pixels)is reasonable to assume for the Gaussian window.Note that the PSF is space-varying since (as given in(2))depends on the depth of the scene for a fixed camera setting.From standard stereo analysis,we know that the depthðDÞis related to the disparity(d),the baseline distanceðbÞ,and the focal length f of the camera.If the focal length of the camera is changed, then for the same depth,the disparity changes.Let d k be the disparity and f k be the focal length associated with the image with blur parameter k,k¼1;2.For stereo[3],we can writed k¼bf kD;k¼1;2:ð4ÞEliminating D,we getd1¼f1f2d2:If we now relax the pin-hole camera model for stereo and substitute the value of depth in terms of disparity,we obtain disparity as a function of blur parameter and camera settings,i.e.,d k¼bf k1kÀ1kÀkk k;k¼1;2:If we assume f k¼V k(since the focal length f k in a pin-hole model is the same as V k in the DFD system),the above equation reduces tod k¼bV kF kÀkr kÀ1;k¼1;2:ð5ÞFrom the above analysis,once the blur is estimated,the disparity can be calculated from the known camera settings.Thus,we can get a dense depth map without explicitly solving the correspondence problem.3D EPTH R ECOVERY AND I MAGE R ESTORATIONIn Section2,we described the geometric relation governing defocus and stereo.In the intensity domain,the relation between the focused and defocused stereo pairs is given by the following observation models:g Lk ¼H k x Lþw Lkg Rk ¼H kðd kÞx Rþw Rk;k¼1;2;ð6Þwhere the image and noise values have been lexicographicallyordered.The noise terms w Lk and w Rkare assumed to beindependent,white Gaussian with zero-mean and variance 2w.The vectors g Lkand g Rkrepresent the observed defocused images with blur parameter k for the k th left and right stereo images, respectively.The blur matrix H k corresponds to the space-variant blurring functionh kði;j;m;nÞ¼12kðm;nÞexpÀðiÀmÞ2þðjÀnÞ22kðm;nÞ!:Note that H kðd kÞis the same as H k with a shift due to disparity. The relation as expressed in(6)matches the left and the right images at the correct location(disparity)and for the correct amount of blurring which is also a function of disparity.By using a single parameter disparity,we have eliminated the(dependent) blur parameter.Since we do not assume local space-invariance,the blur parameter changes with the spatial location.Hence,the matrix H k will not be doubly block-Toeplitz.Given the observation models in(6)and the relations in(5),we attempt to solve for the estimates of depth(blur/disparity)and the focused image of the scene.Note that one needs to estimate only x L(or x R)since they are just shifted versions of one another.From(3),it is clear that we need to estimate either 1ði;jÞor 2ði;jÞ;8i;j.If k is known,the disparity can be calculated from(5).The problem of recovering the focused image and the space-variant blur parameter given two defocused stereo pairs is ill-posed and may not yield a unique solution,unless additional constraints like smoothness are added to restrict the solution space.We model both the space-variant blur parameter and the focused image of the scene as Markov random fields(MRFs).The concept of modeling depth/image as an MRF is well-known in the literature[17].Since the change in the depth of a scene is usually gradual,the space-variant blur parameter also tends to have local dependencies.The utility of the MRF model lies in its ability to capture local dependencies and in its equivalence to the Gibbs random field[18].Moreover,the MRF model preserves locality in the posterior distribution.This helps in reducing the computational complexity substantially.Let S denote the random field corresponding to the space-variant blur parameter 1and X L denote the random field corresponding to the left focused image x L(the intensity process).Let G L1,G L2,G R1, and G R2represent random fields corresponding to the four observed images,g L1,g L2,g R1,and g R2,respectively.The random field S is assumed to be statistically independent of both X L and noise field W Lk.This assumption may not always hold good[19].Let S take P possible levels and X L take M possible levels.Since S and X L are modeled as MRFs,we can writePðS¼sÞ¼1ZexpÀU sðsÞ½ ;ð7ÞPðX L¼x LÞ¼1LexpÀU x Lðx LÞ½ :ð8ÞNote that s is the same as the blur parameter 1.In(7),s has been used for notational consistency.The terms U sð:Þand U x Lð:Þcorre-spond to the energy functions associated with the space-variant blurring process in the left image and the intensity processes in the left image,respectively.Given a realization of S,i.e.,the blur parameter 1ði;jÞ,8i;j,the blurring function h1ðÁ;ÁÞis known and, hence,the matrix H1is also known.Moreover,h2ðÁ;ÁÞand matrix H2 can then be determined because 2ði;jÞ¼ 1ði;jÞþ .Using(5), the disparity can be calculated.Given the four observed images,the a posteriori conditional joint probability of S and X L is given byP S¼s;X L¼x L j G L1¼g L1;G L2¼g L2;G R1¼g R1;G R2¼g R2ðÞ¼P S¼s;X L¼x LðÞP G L1¼g L1;G L2¼g L2;G R1¼g R1;G R2¼g R2j S¼s;X L¼x LðÞP G L1¼g L1;G L2¼g L2;G R1¼g R1;G R2¼g R2ðÞ:ð9ÞFrom Bayes rule,and the independence of S and X L,the problem of simultaneous estimation of blur and focused image can be posed as the following MAP problem:Fig.1.Basic structure of depth from defocused stereo(DFDS).max s;xL PðG L1¼g L1;G L2¼g L2;G R1¼g R1;G R2¼g R2 j S¼s;X L¼x LÞPðS¼sÞPðX L¼x LÞ:From(6)and using the fact that S and X L are statistically independent of each other as well as the noise W Lk,we getÀlog PðG L1¼g L1;G L2¼g L2;G R1¼g R1;G R2¼g R2j S¼s;X L¼x LÞ¼X2 k¼1jj g LkÀH k x L jj22 2wþX2k¼1jj g RkÀH kðd kÞx R jj22 2w:ð10ÞAssuming first-order smoothness for the focused image as well as the blurring process,the posterior energy function to be minimized can be equivalently written asU Pðs;l s i;j;v s i;j;x L;l x L i;j;v x L i;jÞ¼U de f dataþU sm blurþp sm blurþU sm intþp sm intþU st data;ð11Þwhere the horizontal and vertical binary line fields corresponding to the blurring process and the intensity image are denoted by l s i;j,v s i;j, l x L i;j,and v x L i;j,respectively.Line fields have been incorporated into the energy function in order to preserve the discontinuities[18].In(11),the term U de f data corresponds to data fitting based on the defocus information and is given(from(10))asU de f data¼jj g L1ÀH1x L jj22 2wþjj g L2ÀH2x L jj22 2wþjj g R1ÀH1ðd1Þx R jj22 2wþjj g R2ÀH2ðd2Þx R jj22 2w:ð12ÞThe term U sm blur incorporates smoothness for blur and is given as U sm blur¼Xi;js½ðs i;jÀs i;jÀ1Þ2ð1Àv s i;jÞþðs i;jþ1Às i;jÞ2ð1Àv s i;jþ1Þþðs i;jÀs iÀ1;jÞ2ð1Àl s i;jÞþðs iþ1;jÀs i;jÞ2ð1Àl s iþ1;jÞ ;ð13Þwhere s is the regularization parameter corresponding to the blur parameter.Penalty term p sm blur is used to prevent spurious discontinuities and is given byp sm blur¼Xi;js½l s i;jþl s iþ1;jþv s i;jþv s i;jþ1 ;ð14Þwhere s is the associated weight for penalty.Fig.2.(a)Original focused image.(b)Left defocused image with blur 1.(c)Stereo pair of(b).(d)Left defocused image with blur 2.(e)Stereo pair of(d).(f),(g),and (h)Values of 1obtained using DFD alone,stereo alone,and the proposed method,respectively.(i)Space-variant restored image using DFDS.Along similar lines,we have a smoothness term U sm int and a penalty term p sm int for the intensity process also and these are given as follows:U sm int ¼X i;jx ½ðx L i;j Àx L i;j À1Þ2ð1Àv x Li;j Þþðx L i;j þ1Àx L i;j Þ2ð1Àv x L i;j þ1Þþðx L i;j Àx L i À1;j Þ2ð1Àl x L i;j Þþðx L i þ1;j Àx L i;j Þ2ð1Àl x L i þ1;j Þand p sm int ¼Xi;jx ½l x Li;j þl x L i þ1;j þv x L i;j þv x L i;j þ1 ;ð15Þwhere x is the regularization parameter corresponding to theintensity image while x weights the penalty for introducing a discontinuity.Finally,the term U st data corresponds to data fitting due to stereo and is given asU st data ¼ st jj g R 1Àg L 1ðd 1Þjj 2þjj g R 2Àg L 2ðd 2Þjj 2h i;ð16Þwhere the parameter st weights how well the stereo image pairsmatch in terms of disparity.The posterior energy function given by (11)is nonconvex and algorithms based on steepest-descent are prone to get trapped in local minima.We choose the simulated annealing (SA)algorithm [18]for minimizing the posterior energy function so as to obtain estimates of the space-variant blur parameter and the focused image simultaneously.For this purpose,a temperature variable is introduced in the objective function.The cooling schedule was chosen to be linear.Since the random fields associated with SV blur and image are assumed to be statistically independent,the values of blur s i;j and image x i;j at every location ði;j Þare changed independently.Parameter estimation in MRF is a difficult task.Here,we choose the MRF parameters in an ad hoc way.From (11),it may be noted that the estimates of depth and the focused image of the scene using DFD alone can be obtained by simply leaving out the stereo terms [9].However,as we shall show in the next section,the additional constraints provided by stereo are very useful in refining the estimates of depth.4E XPERIMENTAL R ESULTSWe demonstrate the performance of the proposed method in estimating blur (or disparity or depth)and restoring the focused image.The number of discrete levels for the blur parameter was chosen to be 64.For the intensity process,256levels were used (same as the CCD dynamic range).For DFD and DFDS,the window-based method of Subbarao [7]was used to obtain initial estimates of 1.The method of Roy and Cox [20]and the implementation available at http://www2.iro.umontreal.ca/~roys/publi/iccv98/code.html was used to get the stereo estimates.In the first experiment,defocused versions of a random dot-pattern image (Fig.2a)were generated.The blurring was stair-case type and we chose 2ði;j Þ¼0:5 1ði;j Þ.Figs.2b,2c,2d,and 2e show the two defocused stereo pairs of images thus generated.Fig.2f shows the initial estimates of the blur 1using DFD alone (i.e.,without the stereo constraints).Although one can make out the stair-case nature of the blur,the estimates are not very satisfactory.The rms value of the error in the estimate of the blur was found to be 0:55.Since the blur parameter and disparity are related,estimates of the blur obtained using focused left-right stereo pairs are shown in Fig.2g.The rms value of the error is 0:38.The proposed method was next used to estimate the blur/disparity and the original focused image.The values of the various parameters used in the SA algorithm were as follows:T 0¼10:0, s ¼5000:0, f ¼0:005, st ¼0:01, s ¼10:0, f ¼15:0, s ¼0:4, f ¼25:0, s ¼0:1, f ¼6:0,number of annealing iterations ¼200,and the number of metropolis iterations ¼100.Here,T 0is the initial temperature,while s and f are thresholds for deciding the presence of an edge in the blur and in the image,respectively.The variances s 2and f 2are used in a Gaussian sampler to generate new samples of blur and intensity values and these are then used in the SA algorithm togenerate a new realization.The estimated SV blur parameter and the restored image using the proposed method are shown in Figs.2h and 2i,respectively.From the figure,it can be seen that the blur is well-captured and the edges are sharper.The improvement is particularly significant in regions of large blur where defocus is known to perform poorly due to reduced spectral content.This is also reflected in the rms error which now reduces to 0:12.Also,the restored image (which is an estimate of the original image in Fig.2a)is quite good.The method was next tested on the corridor image (obtained from CIL/CMU database)and the defocused stereo image pairs are given in Figs.3a,3b,3c,and 3d.The defocused images were obtained from the ground-truthed disparity and depth values in conjunction with an appropriate choice of camera parameters.The true depth values are given in Fig.3e.The error in the estimates of depth using the DFD method is shown in Fig.3f.A darker gray level implies smaller error.The restored image is shown in Fig.3i.The normalized rms error for the depth map using DFD is 0:209.We note that the estimates are poor at places of large blur where there is not enough spectral content.Note that the ball is quite severely blurred and,hence,the depth estimates in that region are not good for the DFD method.Depth estimates were next obtained using focused stereo pairs of the corridor image.The error in the estimates of depthFig.3.(a)Left defocused image.(b)Right defocused stereo image with the same camera settings as in (a).(c)Left defocused image with different camera parameters.(d)Right defocused stereo pair with same camera settings as in (c).(e)True values of depth.(f)Error in the estimated values of depth using the DFD scheme.(g)Error in the estimated values of depth using only stereo.(h)Error in the estimated values of depth using the proposed scheme.(i)Reconstructed image using DFD alone.(j)Reconstructed image using defocus as well as stereo cues.are displayed in Fig.3g and the normalized rms error is0:264. Results corresponding to the proposed scheme are shown in Figs.3h and3j.By fusing defocus and stereo cues,the depth estimates clearly improve.The error values are smaller compared to both DFD and stereo.We note that the estimates of depth near the cone and the ball are quite good.The rms value of the error in the estimate of depth reduces to0:149.Because the depth estimates for DFDS are better, the restored image is also cleaner with few artifacts.The ball is quite focused and emerges nicely in the restored image(Fig.3j)using DFDS as compared to DFD(Fig.3i).Finally,the performance of the proposed scheme was tested on images captured in our laboratory.The left and right defocused stereo pairs are shown in Figs.4a,4b,4c,ing DFD alone, the depth map and the focused image were first estimated and these are shown in Figs.4e and4g,respectively.From the figures,we note that although the recovered image is reasonably good,the depth map is not satisfactory.The estimates of depth and the focused image using the proposed scheme are given in Figs.4f and4h, respectively.The improvement due to fusion of defocus and stereo is amply evident.Some of the visible artifacts that were present in the restored image using DFD alone are now completely gone (particularly at the end nearer to the camera).The restored image is uniformly focused everywhere and is of very good quality.Also,the planar nature of the variation in the depth of the scene is better brought out by the proposed method.Since a real aperture camera cannot bring all points in a3D scene simultaneously into focus,we do not have focused image pairs of the scene for computing depth using stereo alone.A major significance of our work lies in the following fact:Due to the physics of the problem,no real aperture camera can yield the image that the proposed method has been able to produce in Fig.4h.In effect,Fig.4h is a synthesis of the focused image of the scene had the camera brought the entire3D scene into focus(this is not practically possible since a real aperture camera can only bring points at a single depth into focus).5C ONCLUSIONSWe have proposed a new method for estimating depth that combines defocus and stereo cues for images captured with a real aperture camera.The method uses ideas from both DFD and stereo to its advantage.The estimates of depth are superior compared to both DFD and stereo.The recovered depth map is dense and no feature matching or interpolation is required.The method also simultaneously restores a focused image of the scene.A CKNOWLEDGMENTSS.Chaudhuri gratefully acknowledges the partial support received under the Swarnajayanti Fellowship scheme.R EFERENCES[1] B.Julesz,“Binocular Depth Perception without Familiarity Cues,”Science,vol.145,pp.356-362,1964.[2] A.P.Pentland,“A New Sense for Depth of Field,IEEE Trans.PatternAnalysis and Machine Intelligence,vol.9,pp.523-531,1987.[3] D.Marr and T.Poggio,“Co-Operative Computation of Stereo Disparity,”Science,vol.194,pp.283-287,1976.[4]S.T.Barnard and W.B.Thompson,“Disparity Analysis of Images,”IEEETrans.Pattern Analysis and Machine Intelligence,vol.2,pp.333-339,1980. [5] D.Scharstein and R.Szeliski,“A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,”Int’l puter Vision,vol.47, pp.7-42,2002.[6]M.Subbarao,“Parallel Depth Recovery by Changing Camera Parameters,”Proc.IEEE Int’l puter Vision,pp.149-155,1988.[7]Y.Xiong and S.A.Shafer,“Depth from Focusing and Defocusing,”Proc.IEEE Int’l puter Vision and Pattern Recognition,pp.68-73,1993. [8] A.N.Rajagopalan and S.Chaudhuri,“Space-Variant Approaches toRecovery of Depth from Defocused Images,”Computer Vision and Image Understanding,vol.68,pp.309-329,1997.[9] A.N.Rajagopalan and S.Chaudhuri,“Optimal Recovery of Depth fromDefocused Images Using an MRF Model,”Proc.IEEE Int’l puter Vision,pp.1047-1052,1998.[10]M.Watanabe and S.K.Nayar,“Rational Filters for Passive Depth fromDefocus,”Int’l puter Vision,vol.27,pp.203-225,1998.[11]Y.Y.Schechner and N.Kiryati,“Depth from Defocus vs.Stereo:HowDifferent Really Are They?”Int’l puter Vision,vol.39,pp.141-162, 2000.[12]J.Flusser and T.Suk,“Degraded Image Analysis:An Invariant Approach,”IEEE Trans.Pattern Analysis and Machine Intelligence,vol.20,pp.590-603, 1998.[13] A.Kubota,K.Kodama,and K.Aizawa,“Registration and Blur EstimationMethods for Multiple Differently Focused Images,”Proc.IEEE Int’l Conf.Image Processing,vol.2,pp.447-451,1999.[14]J.Flusser and B.Zitova,“Combined Invariants to Linear Filtering andRotation,”Int’l J.Pattern Recognition and Artificial Intelligence,vol.13, pp.1123-1136,1999.[15] F.Deschenes,P.Fuchs,and D.Zioul,“Homotopy-Based Estimation ofDepth Cues in Spatial Domain,”Proc.IEEE Int’l Conf.Pattern Recognition, vol.3,pp.627-630,2002.[16]Y.P.Tsai,J.Leu,and C.H.Chen,“Depth Estimation by Integration ofDepth-from-Defocus and Stereo Vision,”Inst.of Information Science, Taipei,1998.[17]J.Subrahmonia,Y.P.Hung,and D.B.Cooper,“Model-Based Segmentationand Estimation of3D Surfaces from Two or More Intensity Images Using Markov Random Fields,”Proc.IEEE Int’l Conf.Pattern Recognition,pp.390-397,1990.[18]S.Geman and D.Geman,“Stochastic Relaxation,Gibbs Distributions andthe Bayesian Distribution of Images,”IEEE Trans.Pattern Analysis and Machine Intelligence,vol.6,pp.721-741,1984.[19] D.Rajan and S.Chaudhuri,“Simultaneous Estimation of Super-ResolvedScene and depth Map from Low Resolution Observations,”IEEE Trans.Pattern Analysis and Machine Intelligence,vol.25,pp.1102-1117,2003. [20]S.Roy and I.J.Cox,“A Maximum-Flow Formulation of the N-CameraStereo Correspondence Problem,”Proc.IEEE Int’l puter Vision, pp.492-499,1998.Fig.4.(a)Left defocused image.(b)Stereo pair of(a).(c)Left defocused image with different camera settings.(d)Stereo pair of(c).(e)Estimated values of depth using only DFD.(f)Estimated values of depth using the proposed scheme.(g)Reconstructed image using DFD.(h)Restored focused image of the scene using the proposed method.。