A New Multi-focus Image Fusion Method Using Principal

A New Multi-focus Image Fusion Method Using PrincipalComponent Analysis in Shearlet DomainBiswajit BiswasDepartment of ComputerScience and EngineeringUniversity Of Calcutta,Kolkata,India biswajit.cu.08@Ritamshirsa ChoudhuriDepartment of ComputerScience and EngineeringUniversity Of Calcutta,Kolkata,Indiaritam.shirsa@Kashi Nath DeyDepartment of ComputerScience and EngineeringUniversity Of Calcutta,Kolkata,Indiakndey55@ Amlan ChakrabartiAKCSITUniversity Of Calcutta,Kolkata,Indiaacakcs@caluniv.ac.inABSTRACTThe multi-focus image fusion used to produce a single im-age where the entire view is focused by combining multi-ple images taken with different focus distances.Here we present a concept for multi-focus image fusion using Princi-pal component analysis(PCA)method on shearlet domain. Our proposed concept works on two folds,i)transform the source image into shearlet-image by using shearlet transform (ST),ii)use of PCA model in low-pass sub-band by which the best pixels in smooth parts are selected according to their arrangement.The composition of different high-pass sub-band coefficients achieved by the ST decomposition are realized.Then,the resultant fusion image is reconstructed by performing the inverse shearlet transform(IST).The ex-perimental results,show that our proposed technique can tender enhanced fusion results than some existing methods in the state-of-the-art.This comparative assessment done in the lights of qualitative and quantitative measurements in terms of mutual information and fusion matrix Q AB/F. Categories and Subject Descriptors1.4[Computing Methodologies]:Image Processing—Im-age FusionKeywordsMulti-focus image fusion,shearlet transform,Principal com-ponent analysis,mutual information1.INTRODUCTIONImage fusion is the process of combining information from Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on thefirst page.To copy otherwise,to republish,to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.Request permissions from Permissions@. PerMin’15February26-27,2015,Kolkata,West Bengal,India Copyright2015ACM978-1-4503-2002-3/15/02...$15.00/10.1145/2708463.2709064.two or more images of a scene into a single composite im-age that is more informative and is more suitable for visual perception or computer processing[1].This technology has been widely used in the different areas such as military,re-mote sensing,medicine etc[2].Now-a-days,many image fusion techniques have been proposed to fuse multi-focus images[8,9,11,12].In general,these fusion techniques can be categorized into two classes:i)spatial domain methods ii)transform domain methods[1,2].In spatial domain methods,images can directly fuse inten-sity values of pixels from the input images.The simplest fu-sion method of this class is to compute the average of all the input images,i.e.‘Average scheme’,but average scheme is often suffers sharpness and contrast degradation problem[3]. An improvement over this baseline method is to use for the sharper pixels at each pixel location over all the input im-ages to compose the fused image.This method requires an evaluation metric,such as variance[2],energy of image gra-dient[3],energy of Laplacian[3,5],to establish what amount of a pixel be fused.However,these pixel-based methods e.g.Principle component analysis(PCA),Intensity-Hue-Saturation(IHS),etc.,the presence of noise can cause in-accurate measurement of sharpness and degrade fusion per-formance,while sharpness is evaluated locally around each pixel[3].To resolve the noise issue,block-based methods have been proposed[8].For block based methods,the input images arefirst divided into blocks/regions,the fused image is then composed by selecting the sharper blocks/regions from the input images[12].These block-based methods may suffer that the artifacts can be appear at the boundary of blocks which greatly reduce the quality of the fused image. These limitations of spatial domain methods handled on transform domain methods.The Wavelet transform,Lapla-cian pyramid or Curvelet transform,are pioneering methods of transform domain.These methods produce better results than any spatial domain methods.In recent years,wavelet transform for time-frequency localization,multi-scale char-acteristic and spare representation of target function with point singularity is widely used in image processing and achieved good effect[6,7,10].However,for images con-tained higher dimension singularity,wavelet transform can-not achieve the optimal spare approach[13,14].Thus,toresolve the such limitation,wavelet transform represents im-ages,multi-scale geometric analysis theory is developed and proposed a series of new multi-scale geometric transform method,such as,ridgelet [13],curvelet [13],contourlet [14].Presently,many author have developed fundamental theory of shearlet transform over the affine system that is provided the complete characteristic of analysis and synthesis.The principal component analysis (PCA)is a popular method for feature extraction and dimension reduction and is employed for image fusion [2].Usually,principal compo-nent analysis is a mathematical that transform a number of correlated variables into a several uncorrelated variables.PCA is widely used in data classification [4].We propose image fusion algorithm by combining shearlet transform and PCA techniques and carry out the quality analysis of pro-posed fusion algorithm on set of benchmark Multi-focus im-age.The fusion using PCA is achieved by weighted sum of source images.The weights for each source image are ob-tained from the normalized Eigen vector of the covariance matrices of each source image.Experimental results show that the proposed algorithm provides better results that help to facilitate more accurate analysis of multi-focus images.Our contributions are as follows:•Multi-focus image fusion algorithm using PCA in shear-let domain have been proposed.•In quality analysis,proposed fusion algorithm com-pared with five popular fusion schemes and illustrated that proposed approach more efficient for Multi-focus images.The rest of the paper is organized as follows.in section 2,Shearlet transform have been discussed in brief and Our proposed method is presented in Section 3and the exper-imental results are illustrated in Section 4.In section 5,furnished the conclusion of this paper.2.SHEARLET TRANSFORMConventionally,the shearlet transform was developed based on an affine system with composite dilations [15].The shift-invariant shearlet transform primarily consists of two steps:multi-scale decomposition and directional localization.To save space,we like better to refer the readers to the litera-tures [15,16]for more details.Let us briefly described the continuous and discrete shearlet transform at fixed resolu-tion level j on later subsections:2.1Continuous Shearlet SystemsIn dimension n =2,the affine systems with composite dilations are the collections of the form:ΨP Q (ψ)={ψj,k,l (X )(1)=|det P |j/2ψQ l P jX −k:j,l ∈Z ,k ∈Z 2}where,P,Q are 2×2invertible matrices,f ∈L 2 R 2,conventionally defined as follows:j,k,l| f,ψj,k,l |2=||f ||2(2)The elements of this method are called composite wavelets if ΨP Q (ψ)forms a Parseval frame or tight frame for f ∈L 2 R 2 ,In this system,the dilations matrices P j are relatedwith scale transformations,while the matrices Q l are related with area-preserving geometric transformations,such as ro-tations and shear.The definition of continuous shearlet transform,usually we have,ψa,s,k (x )=a −3/4ψ U −1s V −1a (x −k )(3)Where V a =a 0√a ,U s = 1001 ,ψ∈L 2 R 2 ,following conditions :1.ˆψ( )=ˆψ( 1, 2)=ˆψ1( 1)ˆψ2( 2/ 1);2.ˆψ1∈C ∞(R )supp ψ1⊂[−2,−1/2] [1/2,2],where,ψ1is continuous wavelet;3.ˆψ2∈C ∞(R )supp ψ2⊂[−1,1],ˆψ2>0But ψ2 =1For ψa,s,k ,a ∈R +,s ∈R +and k ∈R +,for any f ∈L 2(R 2),is called shearlet [15],a collection of wavelets with different scales.Here,the anisotropic expansion matrix U s is associated with the scale transform and the shear matrix V a denotes the geometric transformation.Generally,a =4and s =1.Where a,s,k are denote with scale transformations,the shear direction and the translation vector,respectively.2.2The Discrete Shearlet TransformThe process of the discrete shear transformation can be divided into two steps:i)multi-scale subdivision ii)direction localization [15].The figure 1illustrates the decomposition process with the shearlets and the figure 4illustrate shearlet transform of an image (‘Book’).In this process,at any scale j,let f ∈L (Z 2N ).Firstly the Laplacian pyramid method is used to decompose a im-age f j −1a into low-pass image f ja and a high-pass image f j hwith N j =N j −1/4,where f j −1a ∈L (Z 2N j −1),f j a ∈L (Z 2N j )and f j h ∈L (Z 2N j −1).After decomposition,estimated ˆf j b on a pseudo-polar grid with the sub-sequent one-dimensional band pass filter with respect to the signal components,thatgenerate a special matrix D ˆf j b.Then used a band-pass filter on matrix D ˆf j b to reconstruct the Cartesian sampled val-ues and finally performed the inverse two-dimensional FastFourier Transform (FFT)for reconstruct the image.Figure 2illustrate the structure of the frequency tiling provide by the shearlet transform [15].3.THE PROPOSED FUSION ALGORITHMIn this paper,we develop a novel multi-focus image fusionscheme to share the merits of ST and PCA technique.For simplicity,we name this fusion algorithm as ST-PCA fusion algorithm.The main steps of the proposed ST-PCA fusion algorithm is shown in Figure.3.The proposed fusion scheme (ST-PCA)consists two pro-cessing parts:The low frequency coefficients of the original image manipulated by using PCA and integrated them with average method.The high frequency coefficients updated by using an adaptive parameter that is derived from high-pass sub-bands of same and different levels.To summarize the necessary process are described as follows:3.1PCA transform fusion approachPrincipal component analysis is transforms a number of correlated variables into a several uncorrelated variables [5].Figure 1:The figure shows the order of decomposition and directionalfiltering.(a)(b)Figure 2:The structure of the frequency tiling by the shear-let:(a)The tiling of the frequency plane R 2induced by the shearlet,(b)The size of the frequency support of a shearlet ψj,l,k..Figure 3:Schematic diagram of ST-PCA-based fusion algo-rithm.Mostly,the image fusion using PCA is achieved by weighted factor of source images.The weights for each source im-age are obtained from the normalized Eigen vector of the covariance matrices of each source image [2,5].The concept and method of this way is explained in detail in references [2,5],the basics of PCA fusion as follows:•First,the Multi-focus image is transformed with PCA technique and the Eigen values and correspondingEigen-(a)Originalimage (b)Shearletcoefficients(c)Shearlet (d)Re-constructedFigure 4:Illustration of Shearlet transform on Benchmark image Book:(a)Original ‘Book’image,(b)Shearlet coeffi-cients,(c)Shearlet,and (d)Re-constructed ‘Book’image.vectors of correlation matrix between the Multi-focus source images.However,the individual bands is prin-ciple components of derived matrix [4].•Second,the Multi-focus images are matched by the estimated principle components and using as weighted average for individual bands.•Finally,the first band of the first Multi-focus image is multiply with the first principle component and first band of the second Multi-focus image is multiply with the second principle component and finally,for integra-tion of low-pass sub-bands inverse shearlet transform have been done.3.2Evaluation of low frequency sub-band co-efficientsConventionally,the shearlet low-pass sub-bands efficiently represents the approximation of an original image [15].In this paper,the section scheme based on threshold has been used to produce the best fused low-pass coefficients.As,local image features have a highly related with the ST co-efficients,and its corresponding neighborhoods [15].Also,the threshold is determined from the ST coefficients,and its neighborhood.Furthermore,the local disparity reveals of shearlet coefficients is mostly related with the clarity of fused image.Thus,to obtain better fused result,here prin-cipal ST coefficients are selected and combined by principal component analysis (PCA)method.We can summarize the overall procedure as follows:Let L g (i,j )denote the low-pass sub-band located at (i,j ),g =X,Y .The fused low-pass coefficients L F (i,j )are achievedby the following calculation:L F (i,j )=αL X (i,j )+βL Y (i,j )(4)where α,β,α+β=1are two parameters,which are deter-mine via PCA method as follows:1.Initially,we estimate covariance matrix C (i,j )from L g (i,j )and calculate Eigen value V and vector D from matrix C (i,j )successively [4].ter,we have [V,D ]=eigen(C (i,j )),thus if D (i,i )≥D (i +1,i +1),then normalized the i th column of eigen value matrix V otherwise normalized the (i +1)th col-umn of eigen value matrix V .For i =1,we can write as V (:,i )/ n i =1V (:,i )and V (:,i +1)/ ni =1V (:,i +1)respectively.3.Finally,first element of normalized Eigen vale matrix V represents by αand second value by β.that are α=V (:,i )/n i =1V (:,i )and β=V (:,i +1)/ n i =1V (:,i +1)respectively.3.3Evaluation of high frequency sub-band co-efficientsHigh-pass sub-bands of ST provides the details informa-tion of the image,such as edge,corner and etc.[16].To enhance the performance of image fusion method,a new de-cision mapping scheme has been proposed and integrated with shearlets high-pass sub-bands.Let H l,kg (i,j )be the high-pass coefficient at the location (i,j )in the l th sub-band at the k th level,g =X,Y .In short,the process of high-pass shearlet sub-bands by proposed fu-sion method ST-PCA is as follows:For the current sub-band H l,kg of the horizontal levels,letR g,h is the total of H l,kg and the other horizontal sub-bands H m,k g in same level k .It is computed by:R g,h=K k =1L l =1| H l −1,k g −H l,kg |(5)where |·|is absolute distance between two consecutive levels.Similarly,for vertical level,let R g,v is the sum of H l,kgand all the other vertical high-pass sub-bands H m,ng in the different high-pass levels.We evaluated as follows:R g,v =K l,m =1L k,n =1|H l,kg−H m,n g|(6)To determine the horizontal ζg,h and the vertical ζg,v high-pass sub-band dependency factor for the present high-passshearlet sub-band H l,kg ,we performed:ζg,h =R g,hG,h g,v (7)ζg,v =R g,v(R g,h +R g,v )(8)The parameter ζg,h is a relationship between H l,kg and other neighbor high-pass sub-bands in the same horizontal plane,and ζg,v relationship between H l,kg in the different vertical plane with its corresponding neighbor sub-bands.Table 1:Statistical result analysis of ST-PCA Performance metric‘Book ’‘Clock’‘Pepsi ’MI 8.2478.0358.011QAB/F 0.7030.7310.771Finally,to derived the new coefficients H l,kg,new from oldshearlet high-pass sub-bands H l,kg by the factors ζg,h and ζg,v ,we evaluated as follow:H l,k g,new =H l,k g ×1+ζ2g,h +ζ2g,v (9)However,the fused coefficients H l,kF (i,j )are obtained by the following estimation:H F (i,j )=H l,k X,new (i,j )if H l,k X,new ≥H l,k Y,newH l,kY,new (i,j )otherwise(10)The procedure of the ST-PCA fusion algorithm is sum-marized as :•The images to be fused must be registered.•These source images are decomposed by ST transformed.•Best low-pass sub-band are estimated by PCA usingEq.4and the final fused image are obtained via IST.•Largest high-pass shearlet coefficients are selected by Eqs.9,10and fused by proposed fusion rules.•Largest directional shearlet sub-bands is selected.•The fused image is reconstructed by IST.4.EXPERIMENTAL ANALYSISIn this section,we assessed the performance of the pro-posed multi-focus image fusion technique (ST-PCA).The proposed ST-PCA and other selected fusion algorithms are analyzed on a set of benchmark image.Here,three of them are presented to demonstrate the performance of the algo-rithm and two of them are shown the comparative analysis.The benchmark image sets are available at as shown in figure.5.In our experiments,we compared the performance of proposed ST-PCA method with five differ-ent fusion schemes such as (1)Principal component analysis (PCA)[2,4](2)Laplacian pyramid technique (LPT)[5](3)Discrete wavelet transform (DWT)[6],(4)Curvelet trans-form (CVT)[13]and (5)Non-sub-sampled counterlet trans-form (NCST)[14]respectively.Our experimental platform is MATLAB R2010b in the PC Intel Quad-Core i33.2GHz CPU and 8GB memory.Here we have selected two most popular fusion metrics such as Mutual Information (MI)[17]and Q AB/F [18],to evaluate the performance of multi-focus fusion.The MI determines how much information is achieved from the in-put images.This is accomplished by accumulating the mu-tual information of the fused image with each of the input images [17].On the other hand,the performance metric,Q AB/F [18],is based on the assumption that the perceptu-ally important information of an image is the high-frequency edge details.This metric compute how much edge informa-tion is transferred from the input images to the fused image.Table2:Statistical results of comparative assessments of various fusion schemesFusion Method Performance metric‘Clock’‘Pepsi’PCA MI7.1277.301Q AB/F0.7080.719 LPT MI7.4187.325Q AB/F0.7120.722 DWT MI7.5077.480Q AB/F0.7190.732 CVT MI7.6847.541Q AB/F0.7200.740 NSCT MI7.9807.897Q AB/F0.7240.759 ST-PCA MI8.0358.011Q AB/F0.7310.771 The experimental results demonstrate that the focus in the source images are properly fused in thefinal images. Figure1show three fusion results andfigure2showfive fused images obtained by different fusion algorithms on the ‘Pepsi’and‘Clock’image sets.The results of MI and Q AB/F are shown in Table1and Table2.Note that the highest score in each row of Tables2is displayed in bold.From Table1and Table2,it can be observed that the ST-PCA method consistently outperforms the otherfive methods in both evaluation metrics on images setfigure5.In addition to the objective evaluation,we also performed a visual com-parison on image set Pepsi as shown infigure6,and Clock as shown infigure6,.Infigures.??,6are illustrated the re-sultant fused images obtained from the PCA[2,5],LPT[5], DWT[6],CT[13],NCST[14],and ST-PCA respectively. From Tables1and2,we observed that the ST-PCA is bet-ter to the otherfive ones in terms of all the performance metrics,as a result the ST-PCA superior than others.Fur-thermore,we also observed that the quality score of the proposed algorithm by MI and Q AB/F are much higher than the ones of other fusion methods,the fact is that the method is more efficient to preserves the salient information of im-age.From the above experiments,we concluded that the proposed ST-PCA persistently outperforms the otherfive methods in image quality measures.5.CONCLUSIONThe Shearlet provides better analytical information about directionality,localization,anisotropy and multi-scale than traditional multi-scale analysis.In this paper,we propose a new multi-focus image fusion algorithm based on shift-invariant shearlet and PCA or ST-PCA model.The main advantage of our proposed ST-PCA method is to determined optimal maps of focus depth of image which is able to im-prove the focus detection accuracy in smooth regions.Sup-ported by the experimental results,we can conclude that,our ST-PCA method tender better results than many other ex-isting state-of-the-art techniques.6.REFERENCES [1]Anish,A.and Jebaseeli,T.J.A survey on multi-focusimage fusion methods,put.Eng.Technol.(IJARCET).1(8).2012.[2]Konstantinos,parison of nine fusiontechniques for very high resolution data,Photogramm.Eng.Remote Sens.74(5).2008.647-659.[3]Wang,Z.Ziou,D.Armenakis,C.Li,D.and Li,Q.Acomparative analysis of image fusion methods,IEEETrans.Geosci.Remote Sens.43(6).2005.1391-1402. [4]Sun,J.F.Jiang,Y.J.and Zeng,S.Y.A study of PCAimage fusion techniques on remote sensing,Proc.SPIE-5985.2005.739-744.[5]Burt,P.J.The pyramid as a structure for efficientcomputation,in:A.Rosenfeld(Ed.),MultiresolutionImage Processing and Analysis,Springer-Verlag.1984.6-35.[6]Pajares,G.de la Cruz,J.M.A wavelet-based imagefusion tutorial,Pattern Recognit.37(9).2004.1855-1872.[7]Li,H Manjunath,B.S.and Mitra,S.K.Multisensorimage fusion using the wavelet transform,Graph.Models Image Process.57(3).1995.235-245.[8]Li,S.and Yang,B.Multifocus image fusion usingregion segmentation and spatial frequency,Image Vision Comput.26(7),2008.971-979.[9]Aslantas,V.and Kurban,R.Fusion of multi-focusimages using differential evolution algorithm,ExpertSyst.37(12).2010.8861-8870.[10]Tian,J.and Chen,L.Multi-focus image fusion usingwavelet-domain statistics,in:201017th IEEEInternational Conference on Image Processing(ICIP).2010.1205-1208.[11]Zhao,H.Shang,Z.Tang,Y.Y.and Fang,B.Multi-focus image fusion based on the neighbor distance, Pattern Recognit.46(3).2013.1002-1011.[12]Huang,W.and Jing,Z.Evaluation of focus measuresin multi-focus image fusion,Pattern Recognit.Lett.28(4).2007.493-500.[13]Ali,F.El-Dokany,I.Saad,A.and El-Samie F.Acurvelet transform approach for the fusion of MR and CT images,Journal of Modern Optics.57.2010.273-286.[14]Do,M.and Vetterli,M.The contourlet transform:anefficient directional multiresolution image representation, IEEE Trans.Image Process.104(12).2005.2091-2106. [15]Lim,W.Q.The Discrete Shearlet Transform:A NewDirectional Transform and Compactly SupportedShearlet Frames,IEEE TRANSACTIONS ON IMAGE PROCESSING.19(5).2010.1166-1180.[16]Easley,bate,D.and Colonna,F.Shearlet-Based Total Variation Diffusion for Denoising, Image Processing,IEEE Transactions on.18(2).2009.260-268.[17]Qu,G.Zhang,D.and Yan,rmation measurefor performance of image fusion,Electron.Lett.38(7).2002.313-315.[18]Xydeas,C.and Petrovic V.Objective image fusionperformance measure,Electron.Lett.36(4).2000.308-309.(a)Right focus(b)Left focus(c)ST-PCA(d)Right focus(e)Left focus(f)ST-PCA(g)Right focus(h)Left focus(i)ST-PCAFigure5:Multi-focus image fusion results of ST-PCA-based method:(a,d,g)Right Focus.(b,e,h)Left focus.(c,f,i)ST-PCA.(a)PCA(b)LPT(c)DWT(d)CVT(e)NSCT(f)ST-PCA(g)PCA(h)LPT(i)DWT(j)CVT(k)NSCT(l)ST-PCA。