Graph cut based image segmentation with connectivity priors
- 格式:pdf
- 大小:2.19 MB
- 文档页数:8
能量函数构图割图像分割是一种重要的图像分析技术,它不仅得到人们广泛的重视和研究,也在实际中得到大量的应用。
近年来,在计算机视觉领域涌现了大量的图像分割算法。
其中基于能量函数的分割算法具有良好的特性,它通过建立数学模型,将分割问题转化成数学寻优问题,能够清楚地描述要解决的问题,而且与求解问题的算法分开。
基于能量函数的分割方法根据能量函数的类型和寻优过程的不同而区分。
通常主要的两大类是:(1)优化一个定义在连续轮廓或连续曲面的函数;(2)优化一个定义在一系列离散变量上的开销函数。
本文重点研究了第一类的水平集模型和第二类的Graphcut模型在图像分割中的应用。
(1)我们提出一个新的Graphcut模型,该模型利用随机森林算法强的学习和分类性能,来构建Graphcut能量函数,以及相应的图结构。
然后通过最大流算法优化我们的模型得到分割结果;(2)对于水平集模型,我们首先针对Chan-Vese模型提出一个避免求解偏微分方程的快速实现模型,该模型利用每次求得的灰度均值来进行演化,不仅运算量大大减少,同时保持了水平集算法的良好拓扑性能。
最后,我们提出一个基于张量场的水平集模型。
一方面,该模型利用张量结构能够分割纹理图像;另一方面,该模型使用一个区域可变项,能够注重局部信息,从而对于灰度不均匀的图像也能得到比较好的分割结果。
通过能量最小化模型解决一个问题包括两个主要步骤:第一步,描述出一个目标函数,它将所有可能解映射到实数集中,并且给出了可能解的好(坏)程度。
一个目标函数通常是对应该问题的不同约束项的累加,这些约束可以是软约束也可以是硬约束。
在本论文中,所有的目标函数将给出了可能结果的好(坏)程度。
我们称这些目标函数为能量函数。
第二步,最小化能量函数。
这通常是非常艰巨的任务。
计算机视觉中的能量函数通常有很多维和许多局部最小。
许多研究者们已经试用过某些一般的最小化方法,例如梯度下降和模拟退火的方法。
前一个方法几乎可以用于所有连续变量的函数中,后一个方法几乎可以用于所有离散变量函数中。
摘要图像分割是把图像划分为有意义的若干区域的图像处理技术,分割技术在辅助医学诊断及运动分析、结构分析等领域都有着重要的研究价值和广泛的应用发展前景。
在阅读大量文献的基础上,本文对图像分割技术的理论基础、发展历程及图像分割方法的热点、难点问题进行了分类综述,对不同分割算法优缺点进行了总结和归纳,并对图像分割的发展趋势进行了初步的展望和预测。
在此基础上,为了对图像分割理论有更直观的认识,本文选取并行边界算法和分水岭算法这两种方法,用MATLAB软件进行了基础的仿真,并对结果进行了分析和总结,本文重点对一些近年来新兴的算法,比如水平集(Level-set)算法、马尔科夫随机场算法(Markov)、模糊算法、遗传算法、数学形态学算法等进行了概略性的探讨,对这些新兴算法的特点、原理、研究动态进行了分析和总结。
关键词:图像分割;边界;区域;水平集;马尔科夫AbstractImage segmentation is an image processing technology that divides the image into a number of regions. Image segmentation has very important significance in supporting medical diagnosis, motion analysis, structural analysis and other fields.Based on recent research, a survey on the theory and development of image segmentation, hot and difficult issues in image segmentation is given in this article. And describes the characteristics of each method as well as their respective advantages and disadvantages in image segmentation .This article introduces and analyzes some basic imaging and image segmentation methods in theory and describes the development trends of medical image segmentation. To have a better understanding of image segmentation, I use MATLAB software to stimulate on images about the parallel edge algorithms and watershed algorithm. And the analysis of the segmentation results is given in the article.This article introduces and analyzes the new algorithms in recent years such as Level-set algorithm, Markov algorithm, Fuzzy algorithm, Genetic algorithm and Morphological algorithm. In this paper, the features, theory and research trends of these algorithms are analyzed and summarized.Keywords: Image segmentation; Border; Area;Level-set;Markov第1章引言1.1 图像分割的背景和重要作用图像是传达信息的一种方式,图像中含有大量的有用信息,理解图像并从图像中抽取信息以用来完成其他工作是数字图像技术中一个重要的应用领域,而理解图像的第一步就是图像的分割。
graph-cut-based详解-回复Graphcut-based详解Graphcut-based是一种基于图论和图割(graph cut)算法的图像分割方法。
图像分割是一种将图像划分为不同区域或物体的过程,是计算机视觉领域中的重要任务之一。
Graphcut-based方法通过图论的思想和图割的技术,能够有效地实现图像分割,并在图像处理和计算机视觉应用中取得了广泛的应用。
图割是一种基于图的分割方法,它将图像分割问题转化为在图的顶点和边上定义能量函数,并通过最小化该能量函数来得到最佳的分割结果。
Graphcut-based方法是基于图割算法的进一步发展和应用,它通过对图像建模并构建图割框架,从而实现图像分割。
下面将详细介绍Graphcut-based方法的步骤和关键技术。
1. 图像建模:首先,需要对图像进行建模,以便能够在图割框架中进行分割操作。
通常,图像建模可以通过使用颜色、纹理、边缘等特征来描述图像。
这些特征可以反映图像中不同区域的差异,并用于划分图像。
2. 图割框架:建模完成后,将图像表示为一个图,其中图的顶点表示图像中的像素,图的边表示像素之间的关系。
通过为图的顶点和边定义能量函数,将图像分割问题转化为求解能量函数最小化的问题。
常用的能量函数包括数据项和平滑项,前者衡量像素与分割结果的适配程度,后者衡量分割结果的平滑程度。
3. 图割算法:求解能量函数最小化的问题是图割方法的核心。
常用的图割算法有最大流最小割(Max Flow Min Cut)算法和Graphcut算法。
其中,最大流最小割算法通过在图中查找一条从源节点到汇节点的路径,使得路径上的边权重之和最小,从而得到图的最小割。
Graphcut算法则通过迭代地将图像分割为两部分,并计算能量函数的最小值,直到能量函数不再发生变化或达到最大迭代次数。
4. 后处理:分割结果通常需要进行后期处理,以进一步提升分割的质量和准确性。
后处理操作可以包括边缘增强、去除孤立像素、填补空洞等,以获得更好的分割效果。
算注语言信IB与电厢China Computer&Communication2020年第23期基于深度学习的图像分割技术分析张影(苏州科技大学电子与信息工程学院,江苏苏州215009)摘要:近年来,深度学习已广泛应用在计算机视觉中,涵盖了图像分割、特征提取以及目标识别等方面,其中图像分割问题一直是一个经典难题。
本文主要对基于深度学习的图像分割技术的方法和研究现状进行了归纳总结,并就深度学习的图像处理技术进行详细讨论,主要从4个角度讨论处理图像分割的方法,最后对图像分割领域的技术发展做了总结。
关键词:深度学习;图像分割;深度网络中图分类号:TP391.4文献标识码:A文章编号:4003-9767(2020)23-068-02Research Review on Image Segmentation Based on Deep LearningZHANG Ying(College of Electronics and Information Engineering,Suzhou University of Science and Technology,Suzhou Jiangsu215009,China) Abstract:In recent years,deep learning has been widely used in computer vision,covering image segmentation,feature extraction and target recognition,among which image segmentation has always been a classic problem.In this paper,the methods and research status of image segmentation technology based on deep learning are summarized,and the image processing technology of deep learning is discussed in detail.The methods of image segmentation are mainly discussed from four aspects.Finally,the development of image segmentation technology is summarized.Keywords:deep learning;image segmentation;deep network0引言在计算机视觉中,图像处理、模式识别和图像识别都是近几年的研究热点,基于深度学习类型的分割有分类定位、目标检测、语义分割等。
图像分割技术研究综述随着科技的快速发展,图像分割技术作为计算机视觉领域的重要分支,已经在众多应用领域中发挥着越来越重要的作用。
本文将对图像分割技术的研究进行综述,包括其发展历程、应用领域、研究成果以及未来研究方向。
图像分割技术是指将图像按照像素或区域进行划分,从而提取出感兴趣的目标或背景的过程。
图像分割技术在信号处理、计算机视觉、机器学习等领域具有重要的应用价值。
例如,在智能交通中,图像分割技术可以用于车辆检测和跟踪;在医学图像分析中,图像分割技术可以用于病灶区域提取和诊断。
根据图像分割技术所采用的方法,可以将其大致分为以下几类:基于阈值的分割、基于区域的分割、基于边缘的分割、基于模型的分割以及基于深度学习的分割。
1、基于阈值的分割是一种简单而又常用的图像分割方法,其基本原理是通过设定一个阈值,将图像的像素值进行分类,从而将图像分割为不同的区域。
基于阈值的分割方法实现简单、运算效率高,但在处理复杂图像时,往往难以选择合适的阈值,导致分割效果不理想。
2、基于区域的分割方法是根据图像像素的灰度或颜色特征,将图像分割为不同的区域。
这类方法通常适用于均匀背景和简单目标的图像,但对于复杂背景和遮挡情况的处理效果较差。
3、基于边缘的分割方法是通过检测图像中的边缘信息,将不同区域之间的边界提取出来,从而实现图像分割。
这类方法对噪声和光照变化较为敏感,需要结合其他方法进行优化。
4、基于模型的分割方法通常是利用数学模型对图像进行拟合,从而将图像中的目标或背景分离出来。
常用的模型包括参数化模型和非参数化模型两类。
这类方法能够处理复杂的图像特征,但对模型的选择和参数调整要求较高。
5、基于深度学习的分割方法是通过训练深度神经网络,实现对图像的自动分割。
这类方法具有强大的特征学习和自适应能力,能够处理各种复杂的图像特征,但在计算复杂度和训练成本方面较高。
近年来,随着人工智能和机器学习技术的快速发展,基于深度学习的图像分割技术在学术研究和实际应用中取得了显著的成果。
规范化切割和图像分割摘要:为解决在视觉上的感知分组的问题,我们提出了一个新的方法。
我们目的是提取图像的总体印象,而不是只集中于局部特征和图像数据的一致性。
我们把图像分割看成一个图形的划分问题,并且提出一个新的分割图形的全球标准,规范化切割。
这一标准衡量了不同组之间的总差异和总相似。
我们发现基于广义特征值问题的一个高效计算技术可以用于优化标准。
我们已经将这种方法应用于静态图像和运动序列,发现结果是令人鼓舞的。
1简介近75年前,韦特海默推出的“格式塔”的方法奠定了感知分组和视觉感知组织的重要性。
我的目的是,分组问题可以通过考虑图(1)所示点的集合而更加明确。
Figure1:H<iw m.3Uiyps?通常人类观察者在这个图中会看到四个对象,一个圆环和内部的一团点以及右侧两个松散的点团。
然而这并不是唯一的分割情况。
有些人认为有三个对象,可以将右侧的两个认为是一个哑铃状的物体。
或者只有两个对象,右侧是一个哑铃状的物体,左侧是一个类似结构的圆形星系。
如果一个人倒行逆施,他可以认为事实上每一个点是一个不同的对象。
这似乎是一个人为的例子,但每一次图像分割都会面临一个相似的问题一将一个图像的区域D划分成子集Di会有许多可能的划分方式(包括极端的将每一个像素认为是一个单独的实体)。
我们怎样挑选“最正确”的呢?我们相信贝叶斯的观点是合适的,即一个人想要在较早的世界知识背景下找到最合理的解释。
当然,困难在于具体说明较早的世界知识一一些低层次的,例如亮度,颜色,质地或运行的一致性,但是关于物体对称或对象模型的中高层次的知识是同等重要的。
这些表明基于低层次线索的图像分割不能够也不应该旨在产生一个完整的最终的正确的分割。
目标应该是利用低层次的亮度,颜色,质地,或运动属性的一致性继续的提出分层分区。
中高层次的知识可以用于确认这些分组或者选择更深的关注。
这种关注可能会导致更进一步的再分割或分组。
关键点是图像分割是从大的图像向下进行,而不是像画家首先标示出主要区域,然后再填充细节。
江苏科技大学数字图像处理图像分割——区域生长法专题1 图像分割简介图像分割( image segmentation) 就是把图像分成各具特征的区域并提取出感兴趣目标的技术和过程。
这里特征可以是象素的灰度、颜色、纹理等, 预先定义的目标可以对应单个区域也可以对应多个区域。
图像分割是图像处理到图像分析的关键步骤, 在图像工程中占据重要的位置。
一方面, 它是目标表达的基础, 对特征测量有重要的影响。
另一方面, 因为图像分割及其基于分割的目标表达、特征提取和参数测量等将原始图像转化为更抽象更紧凑的形式, 使得更高层的图像分析和理解成为可能。
图像分割是一种重要的图像处理技术, 它不仅得到人们的广泛重视和研究, 在实际中也得到大量的应用。
图像分割包括目标轮廓、阈值化、图像区分或求差、目标检测、目标识别、目标跟踪等技术。
从大的方面来说,图像分割方法可大致分为基于区域的方法、基于边缘的方法、区域与边缘相结合的方法,以及在此基础上的采用多分辨率图像处理理论的多尺度分割方法。
其中基于区域的方法采用某种准则,直接将图像划分为多个区域。
而基于边缘的方法则通过检测包含不同区域的边缘,获得关于各区域的边界轮廓描述,达到图像分割的目的,而区域与边缘相结合的方法通过区域分割与边缘检测的相互作用,得到分割结果。
图像分割中基于区域的方法主要有直方图门限法、区域生长法、基于图像的随机场模型法、松弛标记区域分割法等。
本文主要讨论基于区域分割的区域生长法。
区域生长是一种古老的图像分割方法,最早的区域生长图像分割方法是由Levine等人提出的。
该方法一般有两种方式,一种是先给定图像中要分割的目标物体内的一个小块或者说种子区域,再在种子区域基础上不断将其周围的像素点以一定的规则加入其中,达到最终将代表该物体的所有像素点结合成一个区域的目的;另一种是先将图像分割成很多的一致性较强,如区域内像素灰度值相同的小区域,再按一定的规则将小区域融合成大区域,达到分割图像的目的,典型的区域生长法如T. C. Pong等人提出的基于小面(facet)模型的区域生长法,区域生长法固有的缺点是往往会造成过度分割,即将图像分割成过多的区域。
Graph cut based image segmentation with connectivity priorsSara Vicente∗Vladimir Kolmogorov University College London{s.vicente,vnk}@Carsten Rother Microsoft Research Cambridge carrot@AbstractGraph cut is a popular technique for interactive image segmentation.However,it has certain shortcomings.In particular,graph cut has problems with segmenting thin elongated objects due to the“shrinking bias”.To overcome this problem,we propose to impose an additional connectiv-ity prior,which is a very natural assumption about objects. We formulate several versions of the connectivity constraint and show that the corresponding optimization problems are all NP-hard.For some of these versions we propose two optimization algorithms:(i)a practical heuristic technique which we call DijkstraGC,and(ii)a slow method based on problem de-composition which provides a lower bound on the problem. We use the second technique to verify that for some practi-cal examples DijkstraGC is able tofind the global minimum.1.IntroductionThe task of interactive image segmentation has attracted a significant attention in recent years[10,3,18,6,24,21]. The ultimate goal is to extract an object with as few user in-teractions as possible.It is widely accepted that some prior on segmentations is needed for achieving this goal.Dif-ferent priors have a preference towards different types of shapes,as we discuss next.Graph cut A very popular approach,which we also use in this paper,is based on graph cut[7,3,18].It minimizes an energy function consisting of a data term(computed us-ing color likelihoods of foreground and background)and a spatial coherency term.The latter term is the length of the boundary modulated with the contrast in the image,there-fore minimizing the energy with this term has a bias towards shorter boundaries.(This behavior is sometimes referred to as the“shrinking bias”.)In particular,it is hard for the graph cut approach to segment thin elongated structures.Consider Fig.1.First the user constrains some pixels to be fore-and background using brushes(a).The segmentation by graph cut(b)cuts off some of the legs of the insect.If we re-duce the influence of the coherency term then the legs get(a)User input(b)Graph Cut(GC)(c)GC less coherency(d)Additional input(e)DijkstraGCFigure1.Image segmentation using graph cut with standard(b)and reduced coherency(c)based on input(a).Our new DijkstraGC method(e)with additional user input(d).problems are all NP-hard,as we show.To enable the inter-face shown in Fig.1we propose a heuristic algorithm whichwe call DijkstraGC.On an abstract level it merges the Dijk-stra algorithm and graph cut.Note that Dijkstra-like meth-ods have already been used for extracting thin objects suchas blood vessels[5],although without an explicit segmen-tation.(A fast marching technique was used in[5],whichcan be viewed as a continuous analogue of the Dijkstra al-gorithm for discrete graphs.)The key feature of our methodthat distinguishes it from[5]is the addition of the graph cutcomponent.This allows to explicitly use the MAP-MRFformulation which proved to be very successful[3,18].We show that on some practical examples DijkstraGC isable tofind the global minimum.In order to verify this,wedeveloped a second(slow)technique based on dual decom-position,which provides a lower bound on the problem.Related work Connectivity is automatically enforced inthe classical“snakes”approach[11],since the segmenta-tion is represented by a simple closed contour.Han et al.[9]proposed a topology preserving level set method which al-lows to specify more general topologies.A disadvantage ofboth techniques is that the objective is optimized via gradi-ent descent,which can easily get stuck in a local minimum.Recently,Zeng et al.[29]followed a similar approach witha discrete graph-based formulation.After posing the prob-lem the authors of[29]proved an NP-hardness result andproposed to modify the maxflow algorithm in[4]so that thetopology of the segmentation is preserved.However,de-spite our best effort we were unable to compare it to ourapproach for the task of segmenting thin objects.1(Note,results in[29]are shown for very different types of objects.)2.Problem formulationWe use an energy function of the form which is standardfor graph cut based image segmentation approaches[3,18]:E(x)= p∈V E p(x p)+ (p,q)∈E E pq(x p,x q)(1)Here(V,E)is an undirected graph whose nodes corre-spond to pixels.x p∈{0,1}is the segmentation label ofthen the problem with C1can be reduced to a shortest path computation with a single source and a single sink and thus can be solved in polynomial time(see section3).Enforcing constraint C1may result in a segmentation which has a“width”of one pixel in certain places,which may be undesirable(see Fig.6).One way tofix this prob-lem is to allow the user to specify a parameterδwhich con-trols the minimum“width”of the segmentation.Formally, assume that for each node p∈V we have a subset Q p⊆V. (This subset would depend onδ;for example,for a grid graph Q p could be the set of all pixels q such that the dis-tance from p to q does not exceedδ.)Using these subsets, we define the following connectivity constraint:C2There must exist a path in the graph(V,F)from s to t such that for all nodes p in the path the subset Q p belongs to[x],i.e.x q=1for q∈Q p.Clearly,C1is a special case of C2if we choose Q p={p} for all nodes p.Throughout the paper,we denote P0,P1,P2to be the problems of minimizing function(1)under constraints C0, C1,C2,respectively.The theorem below shows the diffi-culty of the problems;its proof is given in[26]. Theorem1.Problems P0,P1,P2are NP-hard.P0and P2 remain NP-hard even if the set E is empty,i.e.function(1) does not have pairwise terms.Note,it was also shown in[29]that the following prob-lem is NP-hard:minimize function(1)on a planar2D grid so that the foreground is4-connected and the background is 8-connected.It is straightforward to modify the argument in[29]to show that the problem is NP-hard if only the4-connectedness of the foreground is imposed(in other words, P0is NP-hard even for planar2D grids).To conclude this section,we will state some simple facts about the relationship of problems P0-P2and the problem of minimizing function E(x)without any constraints. Theorem2.Suppose that x is a global minimum of func-tion(1)without any constraints.(a)There exists an optimal solution x∗of P2which in-cludes x,i.e.[x]⊆[x∗].The same holds for the problem P1since the latter is a special case.(b)Suppose that E⊆F.Let C1,...,C k⊆V be the con-nected components of the set[x]in the graph(V,F).Then there exists an optimal solution x∗of P0such that each component C i is either entirely included in [x∗]or entirely excluded.In other words,if C i and [x∗]intersect then C i⊆[x∗].A proof is given in[26].The theorem suggests that as afirst step we could run the maxflow algorithm to min-imize function(1)without any constraints and then con-tract connected components of the obtained set[x]to sin-gle nodes.However,it leaves open the most challenginginitialize:S=∅,PARENT(p)=NULL for all nodes p, d(s)=min{E(x)|Q s⊆[x]},d(p)=+∞for p∈V−{s}while t/∈S and V−S contains nodes p with d(p)<+∞•find node p∈V−S with the smallest distance d(p)•add p to S•for all nodes q∈V−S which are neighbors of p(i.e.(p,q)∈F)do-using PARENT pointers,get path P from s to qthrough p;compute corresponding set¯P=∪r∈P Q r -compute a minimum x of function(1)under the con-straint¯P⊆[x]-if d(q)>E(x)set d(q):=E(x),PARENT(q):=pFigure2.DijkstraGC algorithm.question:what to do if a minimum of function(1)does not satisfy the desired connectivity constraint.3.AlgorithmsThe main algorithmic contribution of this paper is a heuristic method for the problem P2(and thus for P1since the latter is a special case).This method,which we call Di-jkstraGC,is presented in section3.1.Then in section3.2we propose an alternative method for a special case of problem P1based on the idea of problem decomposition.The main feature of the second technique is that it provides a lower bound on the optimal value of P1.We will use it for as-sessing the performance of DijkstraGC:in the experimental section it will help us to verify that for some instances Di-jkstraGC gives an optimal solution.3.1.DijkstraGC:Merging Dijkstra and graph cutsThe idea of ourfirst method is motivated by the Dijk-stra algorithm[1].Recall that the latter technique com-putes shortest distances d(p)in a directed graph with non-negative weights from a specified“source”node s to all other nodes p.Similar to the Dijkstra method,we will compute solu-tions to the problem P2for afixed node s and all nodes p∈V(only now these solutions will not necessarily be global minima).The“distance”d(p)will now indicate the cost of the computed solution for the pair of nodes{s,p}.The algorithm is shown in Fig.2.During the algorithm, the current solution x p for node p with d(p)<+∞can be obtained as follows:using PARENT pointers get path P and corresponding set¯P=∪r∈P Q r,and then compute a minimum of function(1)under the constraint¯P⊆[x]. Clearly,the obtained solution x p satisfies the hard con-straint C2for the pair of nodes{s,p}.The set S contains“permanently labeled”nodes:once a node p has been added to S,its cost d(p)and the corre-Q c={c,b,b′},Q t={t,b,b′}Q p={p}for all other nodes p(a)Problem P1(b)Problem P2,no pairwise terms Figure3.Suboptimality of DijkstraGC.Examples of problems on which DijkstraGC give suboptimal results.Graphs shown in the pictures are the connectivity graphs(V,F).Number c p at node p gives the unary term cx p,number c pq at edge(p,q)gives the pairwise term c pq|x q−x p|.Both in(a)and(b)DijkstraGC will output solution{s,a,b,b′,t}or{s,a′,b,b′,t}with cost7,while the optimal solution{s,c,b,b′,t}has cost6.sponding solution will not change anymore.Let us list some of the invariants that are maintained dur-ing DijkstraGC(they follow directly from the description): I1If d(p)=+∞then p=s and PARENT(p)=NULL.I2If d(p)<+∞then PARENT pointers give the unique path P from s to p,and d(p)=min{E(x)|¯P⊆[x]} where¯P=∪r∈P Q r.I3If PARENT(q)=p then d(p)≤d(q)<+∞.I4d(p)<+∞for nodes p∈S.Theorem3.If function E(x)does not have pairwise terms and Q p={p}for all nodes p(i.e we have an instance of P1)then the algorithm in Fig.2produces an optimal solu-tion.A proof is given in[26].After submission we also found another special case in which DijkstraGC gives an optimal result(see[26]).If conditions of the theorem are relaxed then the problem may become NP-hard,as theorem1states.Not surprisingly, DijkstraGC may then produce a suboptimal solution.Two examples are shown in Fig.3.Note that in these examples the“direction”of DijkstraGC matters:if we run it from s to t then we obtain a suboptimal solution,but running DijkstraGC from t to s will give an optimal segmentation.We now turn to the question of efficient implementation. One computational component of the algorithm isfinding a node p∈V−S with the smallest value of d(p)(same as in the Dijkstra algorithm).We used a binary heap struc-ture for implementing the priority queue which stores nodes p∈V−S with d(p)<+∞.The bottleneck,however,is maxflow computations:DijkstraGC requires many calls to the maxflow algorithm for minimizing function(1)under the constraints x r=1for nodes r∈¯P.These computa-tions are considered in the remainder of this section. Optimized DijkstraGC First,we will describe a tech-nique which allows to reduce the number of calls toinitialize:S=∅,PARENT(p)=NULL for all nodes p, d(s)=min{E(x)|Q s⊆[x]},d(p)=+∞for p∈V−{s}while t/∈S and V−S contains nodes p with d(p)<+∞•find node p∈V−S with the smallest distance d(p)•using PARENT pointers,get path P from s to p;com-pute corresponding set¯P=∪r∈P Q r•compute a minimum x of function(1)under the con-straint¯P⊆[x]•add p to S,set A={p},mark p as“unprocessed”•while A has unprocessed nodes-pick unprocessed node p′∈A-for all edges(p′,q)∈F with q∈V−S do⋄if Q q⊆[x]set d(q):=E(x),PARENT(q):=p′,add q to S and to A as an unprocessed node -mark p′as“processed”•for all nodes q∈V−S which are neighbors of A(i.e.(p′,q)∈F for some node p′∈A)do-pick node p′∈A with(p′,q)∈F-using PARENT pointers,get path P from s to qthrough p′;compute corresponding set¯P=∪r∈P Q r -compute a minimum x of function(1)under the con-straint¯P⊆[x]-if d(q)>E(x)set d(q):=E(x),PARENT(q):=p′Figure4.Optimized version of the DijkstraGC algorithm. maxflow.Consider the step that adds node p to the set of permanently labeled nodes S.Denote P to be the path from s to p given by PARENT pointers,and let¯P=∪r∈P Q r. Let usfix nodes in¯P to1and compute a minimum x of function(1)under these constraints.The segmentation set [x]will contain¯P,but it may include many other nodes as well.Then it might be possible to add several nodes to S using this single computation.Indeed,suppose p has a neighbor q∈V−S,(p,q)∈F,such that Q q⊆[x].The algorithm in Fig.2would set d(q)=d(p)=E(x)while exploring neighbors of p.This would make the distance d(q)to be the smallest among nodes in V−S,so the node q could be the next node to be added to S.Therefore,we can add q to S immediately.An algorithm which implements this idea is shown in Fig.4.Before exploring neighbors of q,we check which nodes can be added to S for“free”.The set of these nodes is denoted as A;clearly,it includes p.After adding nodes in A to S,we explore neighbors of A which are still in V−S.Note that there is a certain freedom in implementing the DijkstraGC algorithm:it does not specify which node p∈V−S with the minimum distance to choose if there areseveral such nodes.It is not difficult to see that under a certain selection rule DijkstraGC becomes equivalent to the algorithm in Fig.4.Flow and search tree recycling We used the maxflow algorithm in[4],and reusedflows and search trees as de-scribed in[13].In DijkstraGC we often need tofix/unfix nodes in differ-ent parts of the graph in a rather chaotic order.We believe that this significantly reduces the effectiveness offlow and search tree recycling.Two ideas could potentially be used to overcome this drawback.Thefirst one is based on the ob-servation that different“branches”are often independent in a certain sense.This could allow to reorder maxflow com-putations.To get the same type of result as DijkstraGC we would need to redo computations if we detect an in-consistency,as in the Bellman-Ford label-correcting algo-rithm.The second idea is to maintain multiple graphs for performing computations in different parts of the image, so that changes in each graph would be more“local”.It could also be feasible to store a small subset of the nodes for each graph,increasing it“on demand”.Reduced mem-ory requirements could then allow to use a larger number of graphs.Exploring these ideas is left as a future work.3.2.Problem decomposition approachIn this section we propose a different technique for a spe-cial case of problem P1;we will use it for assessing the performance of DijkstraGC.Overview On the high level,the idea is to decompose the original problem into several“easier”subproblems,for which we can compute efficiently a global minimum(or obtain a good lower bound).Combining the lower bounds for individual subproblems will then provide a lower bound for the original problem.The decomposition and the corre-sponding lower bound will depend on a parameter vectorθ; we will then try tofind a vectorθthat maximizes the bound.This approach is well-known in combinatorial opti-mization;sometimes it is referred to as“dual decomposi-tion”[2].In vision the decomposition approach is probably best known in the context of the MAP-MRF inference task. It was introduced by Wainwright et al.[27]who decom-posed the problem into a convex combination of trees and proposed message passing techniques for optimizing vector θ.These techniques do not necessarilyfind the best lower bound(see[14]or review article[28]).Schlesinger and Giginyak[19,20]and Komodakis et al.[17]proposed to use subgradient techniques[23,2]for MRF optimization, which guarantee to converge to a vectorθyielding the best possible lower bound.Solving P1via problem decomposition We now apply this approach to P1.To get tractable subproblems,we im-pose the following simplifying assumptions.First,we as-sume that the graph(V,F)is planar,and E=F.Sec-ond,we assume that pixels on the image boundary are con-strained to be background,i.e.their label is0.We argue that these assumptions represent an important practical subclass of the image segmentation task,and thus can be used for assessing the performance of DijkstraGC for real problems. Note that the second assumption encodes the prior knowl-edge that the object lies entirely inside the image,which is very often the case in practice.We denote C(x)to be the hard constraint term which is0if the segmentation x satisfies the connectivity con-straint C1and the background boundary condition de-scribed above,and otherwise C(x)is+∞.Some of these hard constraints will also be included in function E(x)as unary terms,namely the background boundary constraints and foreground constraints x s=x t=1,which follow from C1.Our parameter vectorθwill have two parts:θ=(θ1,θ2)where vectorsθ1andθ2correspond to nodes and edges of the graph(V,E),respectively(θ1∈R V,θ2∈R E).Given labeling x,letφ(x)∈{0,1}E be the vector of indicator variables showing discontinuities of x, i.e.φpq(x)=|x q−x p|for an edge(p,q)∈E.We will use the following decomposition:E(x)+C(x)=E0(x|θ)+E1(x|θ)+E2(x|θ)(2) whereE0(x|θ)=E(x)− x,θ1 − φ(x),θ2 (2a)E1(x|θ)=C(x)+ x,θ1 (2b)E2(x|θ)=C(x)+ φ(x),θ2 (2c) Let us discuss each subproblem in more detail. Subproblem0Function E0(x|θ)consists of unary and pairwise terms.We will require this function to be submod-ular;this is equivalent to specifying upper bounds on com-ponentsθ2pq.Since there are no connectivity constraints,we can compute the global minimumΦ0(θ)=min x E0(x|θ) using a maxflow algorithm2.Subproblem1Function E1(x|θ)has only unary terms and the connectivity constraint C1.As discussed in the previous section,we can compute the global minimum Φ1(θ)=min x E1(x|θ)using,e.g.DijkstraGC algorithm. Note,in this case it is essentially equivalent to the Dijkstra algorithm.Subproblem2We will require vectorθ2to be non-negative.We compute a lower boundΦ2(θ)on E2(x|θ2) using a very fast technique whose details are given in[26]. In short,we compute two edge disjoint paths of minimum cost in the dual graph from a set of nodes“behind”node s to a set of nodes“behind”node t.(This is motivated by thefact that an optimal segmentation can be viewed as a simple closed contour going“around”s and t.)Maximizing the lower bound We described a lower bound on problem P1which can be written asΦ(θ)=Φ0(θ)+Φ1(θ)+Φ2(θ)≤E(x)+C(x) whereθbelongs to a convex setΩ={(θ1,θ2|0≤θ2pq≤θ2max pq }.Clearly,Φis a concave function ofθ.Similarto[19,20,17],we used a projected subgradient method[23, 2]for maximizingΦ(θ).Details of our implementation and the procedure for choosing solution x are given in[26]. 4.Experimental resultsIn the previous section we presented DijkstraGC,a new algorithm that minimizes energy(1)under certain connec-tivity constraints on the segmentation x.In this section we first discuss the advantages of including this algorithm in an interactive system for image segmentation and second con-sider the optimality properties of the algorithm.4.1.DijkstraGC for interactive segmentationThe form of the energy(1)follows the approach of pre-vious energy minimization techniques for interactive image segmentation[3,18].We define E p(x p)as a data likelihood term and E pq(x p,x q)as a contrast-dependent coherency term,which are defined as follows.Hard constraints for background and foreground are specified in the form of brush strokes.Based on this input a probabilistic model is computed for the colors of background(G B)and foreground(G F)us-ing two different Gaussian Mixture Models.E p(x p) is then computed as E p(0)=−log(Pr(z p|G B))and E p(1)=−log(Pr(z p|G F))where z p contains the three color channels of site p(see details in[18]).The co-herency term incorporates both an Ising prior and a contrast-dependent component and is computed asE pq(x p,x q)=|x q−x p|(a)User input(b)Graph Cut[18](c)Additional user input(d)DijkstraGC(e)Problem Specificationsize=481×321time=1.0δ=1size=568×426time=2.9δ=2size=640×865time=14.8δ=3Figure5.Results of the DijkstraGC algorithm.(a)original imageswith user scribbles(blue background;green foreground);(b)Graph Cut resultsusing[18];(c)Selection of sites for connectivity,where numbers present the input order;(d)DijkstraGC results;(e)Problem specification:image size,running time for DijkstraGC(on2.16GHz CPU with2GB RAM),and minimum width specified by the user.(a)User input(b)Graph Cut(c)DijkstraGCδ=1(d)DijkstraGCδ=2Figure6.Width parameterδ.Two different results obtained with DijkstraGC algorithm for different values ofδ(minimum width).1%of the number of pixels in set[x])and the difference isoften not visually significant.In contrast,the difference in speed can be substantial.Inour examples the running time was on average reduced byhalf if the“source”node s was in the smaller component(out of the two components that we want to connect).Ac-cordingly,we chose it as the default option and used it forthe results presented in Fig.5and6.4.2.Optimality of DijkstraGCThe dual decomposition algorithm,described in sec-tion3.2,gives both a solution for a special case of P1anda lower bound on the optimal value of P1.Although thistechnique is not useful for a practical system,since the run-ning time is on average3hours,it can be used to assess theoptimality of DijkstraGC.We considered40connectivity problems(er clicks)where the dual decomposition approach is applicable,i.e.all pixels at the image boundary are background.Anotherrestriction for this approach is that we have to use a pla-nar graph(4-connected2D grid)for maxflow computations.For12out of the40problems the dual decomposition algo-rithm gave the global optimum.It is a positive result that forall these12cases also DijkstraGC returned the global opti-mum.Thefirst image in Fig.5is one of the examples for which we obtained the global optimum for all the connec-tivity constraints.(Note that the result is slightly different from the one presented,since for this optimality experiment we had to choose the graph to be planar,i.e.4-connected.) For all the other problems we observed that the result pro-vided by DijkstraGC was always better in terms of energy value than the result of the dual decomposition method. 5.Conclusions and Future WorkIn this paper we proposed to overcome the“shrinking bias”of graph cut methods by imposing connectivity con-straints in the segmentation.We presented a new algorithm DijkstraGC that computes a segmentation satisfying those constraints and we showed that integrating this algorithm in an interactive system for image segmentation reduces con-siderably the amount of user interaction necessary to seg-ment thin structures in the image.Although in general DijkstraGC is not guaranteed to compute the global minimum of our NP-hard optimization problem,we believe that in practice it is not an issue.This claim is supported by two facts:(i)running DijkstraGC in different directions gives almost the same result,and(ii)Di-jkstraGC computes the optimal solution for some particular instances(see sec.4.2).Currently,the speed of DijkstraGC is perhaps the main drawback for a practical interactive segmentation system. However,we believe that there is a large scope for improve-ment via rearrangement of the order in which nodes are visited during the algorithm,or the use of multiple graphs for maxflow computations(sec.3.1).We intend to explore these ideas in the future.References[1]R.Ahuja,T.Magnanti,and work Flows:Theory,Algorithms,and Applications.Prentice Hall,1993.[2] D.Bertsekas.Nonlinear Programming.Athena Scientific,1999.[3]Y.Boykov and M.-P.Jolly.Interactive graph cuts for optimalboundary and region segmentation of objects in N-D images.In ICCV,2001.[4]Y.Boykov and V.Kolmogorov.An experimental comparisonof min-cut/max-flow algorithms for energy minimization in vision.PAMI,26(9),Sept.2004.[5]T.Deschamps and L.D.Cohen.Fast extraction of minimalpaths in3D images and applications to virtual endoscopy.Medical Image Analysis,5(4):281–299,2001.[6]L.Grady.Random walks for image segmentation.PAMI,28(11):1768–1783,Nov.2006.[7] D.Greig,B.Porteous,and A.Seheult.Exact maximum aposteriori estimation for binary images.J.of the Royal Sta-tistical Society,Series B,51(2):271–279,1989.[8]P.L.Hammer,P.Hansen,and B.Simeone.Roof duality,complementation and persistency in quadratic0-1optimiza-tion.Mathematicl Programming,28:121–155,1984.[9]X.Han,C.Xu,and J.L.Prince.A topology preservinglevel set method for geometric deformable models.PAMI, 25(6):755–768,2003.[10]I.Jermyn and H.Ishikawa.Globally optimal regions andboundaries as minimum ratio weight cycles.PAMI,23(10), Oct.2001.[11]M.Kass,A.Witkin,and D.Terzopoulos.Snakes:Activecontour models.IJCV,1(4):321–331,1987.[12]R.Kimmel and A.M.Bruckstein.On regularized Lapla-cian zero crossings and other optimal edge integrators.IJCV, 53(3):225–243,2003.[13]P.Kohli and P.H.S.Torr.Efficiently solving dynamicMarkov randomfields using graph cuts.In ICCV,2005. [14]V.Kolmogorov.Convergent tree-reweighted message pass-ing for energy minimization.PAMI,28(10),2006.[15]V.Kolmogorov and Y.Boykov.What metrics can be approx-imated by geo-cuts,or global optimization of length/area and flux.In ICCV,2005.[16]V.Kolmogorov,Y.Boykov,and C.Rother.Applications ofparametric maxflow in computer vision.In ICCV,2007. [17]N.Komodakis,N.Paragios,and G.Tziritas.MRF optimiza-tion via dual decomposition:Message-passing revisited.In ICCV,2005.[18] C.Rother,V.Kolmogorov,and A.Blake.Grabcut-inter-active foreground extraction using iterated graph cuts.SIG-GRAPH,August2004.[19]M.I.Schlesinger and V.V.Giginyak.Solution to structuralrecognition(MAX,+)-problems by their equivalent transfor-mations.Part1.Control Systems and Computers,(1):3–15, 2007.[20]M.I.Schlesinger and V.V.Giginyak.Solution to structuralrecognition(MAX,+)-problems by their equivalent transfor-mations.Part2.Control Systems and Computers,(2):3–18, 2007.[21]T.Schoenemann and D.Cremers.Introducing curvature intoglobally optimimal image segmentation:Minimum ratio cy-cles on product graphs.In ICCV,Oct.2007.[22]J.Shi and J.Malik.Normalized cuts and image segmenta-tion.PAMI,22(8):888–905,Aug.2000.[23]N.Z.Shor.Minimization methods for nondifferentiable func-tions.Springer-Verlag,1985.[24] A.K.Sinop and L.Grady.A seeded image segmentationframework unifying graph cuts and random walker which yields a new algorithm.In ICCV,Oct.2007.[25] A.Vasilevskiy and K.Siddiqi.Flux maximizing geometricflows.PAMI,24(12),2002.[26]S.Vicente,V.Kolmogorov,and C.Rother.Graph cut basedimage segmentation with connectivity priors.Technical re-port,UCL,2008.[27]M.Wainwright,T.Jaakkola,and A.Willsky.MAP estima-tion via agreement on trees:Message-passing and linear-programming approaches.IEEE rmation Theory, 51(11):3697–3717,2005.[28]T.Werner.A linear programming approach to max-sumproblem:A review.PAMI,29(7),2007.[29]Y.Zeng,D.Samaras,W.Chen,and Q.Peng.Topology cuts:A novel min-cut/max-flow algorithm for topology preservingsegmentation in N-D images.Technical report,2007.。