RBF Neural Networks Optimization Algorithm Based on Support Vector Machine and Its ApplicationREN Jin-xia，YANG Sai(College of Mechanical and Electronic Engineering, JiangXi University of Science and Technology, Ganzhou 341000,China) Abstract: Support vector machine (SVM) resembles RBF neural networks in structure. Considering their resemblance , a new optimization algorithm based on support vector machine and genetic algorithm for RBF neural network is presented，in which GA is used to choose the SVM model parameter and SVM is used to help constructing the RBF. The network based on this algorithm is applied on nonlinear system identification.Simulation results show that the network based on this algorithm has higher precision and better generalization ability.Key words : support vector machines (SVM) ; neural networks ; genetic algorithm; system identificationRadial basis function (RBF) networks is one of new and effective feed-forward neural networks, which can approximate any continuous function with arbitrary precision. RBF networks are always with simple architecture, and trained fast. In RBF networks, the number and place of hidden center vector are key points concerned to the performance of the RBF networks. There exits obvious defectiveness to certain the parameters of RBF networks by the traditional ways.Based on statistical learning theory, support vector machines (SVM) algorithm has a solid mathematical theoretical foundation and rigorous theoretical analysis, which has the advantage of theoretical completeness, global optimization, adaptability, and good generalization ability. It is largely solved the past problems of choosing machine learning model, over-fitting, non-linear, the curse of dimensionality, local minimum points and so on. It uses the structural risk minimization principle, which minimizes the empirical risk; at the same time effectively improve generalization ability of the algorithm.SVM has the structure similarities with RBF network, in this paper, we study the intrinsic link between these two algorithms, and propose RBF optimization algorithm based on SVM and genetic algorithm, which use GA to choose the parameters for SVM, then to be used for structuring RBF networks. This algorithm effectively improves generalization and don’t need a large number of experiments or empirical experiences to pre-specify network structure.1.SVM providing theoretical foundation for structure and parameters of RBFRadial basis function (RBF) networks typically have three layers: an input layer, a hidden layer with a non-linear RBF activation function and a linear output layer. RBF network from input to output mapping is non-linear, but in terms of the weights of network output is linear. The k th hidden unit’s output is:22()exp2kkkx cxφσ⎛⎫-= -⎪⎪⎝⎭(1) Where ⋅is Euclidean norm, i x is the i th inputvector,kc is the center vector of hidden units,kσis the width of hidden units. N denotes thenumber of the hidden units,kw is the weights between the hidden units and the outputs, thenthe outputs of RBF networks is :()1221()exp 2Nk k k N kk k k f x w x x c w φσ===⎛⎫-=⨯ -⎪ ⎪⎝⎭∑∑ (2)According to Mercer Conditions, SVM adopts kernel function to map a sample vector from the original space to feature space. Gaussian kernel function used here is :()22,exp 2ii i x v K x v σ⎛⎫-= -⎪ ⎪⎝⎭(.3) SVM in regression form is the linear combination ofthe hidden units, then:1221()(,)exp 2gi i i gii i i f x w K x v bx v w b σ===+⎛⎫-=⨯ -⎪+ ⎪⎝⎭∑∑ (4)SVM network architecture can be indicated as shown in Figure1,which is similar to a RBF network.Figure 1 the structure of SVMSVM has a similarity in structure with the RBF network, so the number g of support vector which is gotten from the training of SVM can be the number of the hidden units in RBF networks, support vector can be the center vector of radius function, the width selected by SVM can be the width of RBF.2. GA providing SVM modelsparametersIn designing SVM, the parameterσ, penaltyfactor C, insensitive loss function εare the important factors to affect the performance of SVM, especiallyin small samples learning condition, so selectingproperσ、C and εwill be very important to thelearning machine performances. We will use the most widely used genetic algorithm to select those three parameters.The algorithm is to use genetic algorithm to optimize the SVM model parameters, the basic steps of it are as follows:(1)Choose the initial population of individuals randomly;(2)Evaluate the fitness of each individual in that population;(3)Select a new generation of population from the previous generation by using selection operator; (4)Take the crossover and mutation operation on the current population, then take the evaluation, selection, crossover and mutation operation on the new breed, and continue.(5)If the fitness function value of optimal individual is large enough or the algorithm have run many generations, and the optimal fitness value of theindividual can ’t be changed obviously, then we get the optimal value of kernel function parameter σ, penalty factor C , and insensitive loss function ε,and we can also get the optimal classifier by the training datum. How to construct fitness function is the key point of genetic algorithm. We use the promotion theorem of SVM in high-dimensional space to construct fitness function. We set()1/0.01fit T =+ (5) Where 221R T l γ=is the testing error boundary ，R is the radius of super-sphere which contain allthe datum,1/w γ=is the interval value ,l isthe number of the samples.3 SVM providing networks structure and parameters for RBFFor support vector regression machine, its learningprocess can be concluded to be a quadratic programming problem under the linear constrains, then we can use the regression machine which have been trained well to certain the structure and parameters of RBF networks.First considering the linear regression conditions, the sample are 1122(,),(,),...,(,)n n n x y x y x y R R ∈⨯, the linear function is set to be ()f x w x b =⋅+,then the optimization problem will be a minimization problem.11(,,)()2ni i i R w w w C ξξξξ**==⋅++∑(6)The conditions are :(),1,...,(),1,...,,0,1,...,i i i i i i i i f x y i n y f x i ni nξεξεξξ**-≤+=-≤+=≥= (7)Introducing Lagrange function, we can get: 1111(,,,,,,,)1()()[()]2[()]()n ni i i i i i i i n niii i i i i i i i L w b w w C w x b y w x b y ξξααηηξξαεξαεξηξηξ****==****===⋅++-++⋅+--+-⋅-+-+∑∑∑∑ (8) Where,0,,0,1,...,i i i i i n ααηη**≥≥=.Solving this quadratic optimization problem, we can get ,i i αα*, then 1()ni i i i w x αα*==-∑,we can also get bbyi i i i b y w x b y w x εε=-⋅-⎧⎨=-⋅+⎩(9) So the regression function is1()()()()ni i i i f x w x b x x b αα*==⋅+=-⋅+∑ (10)Non-linear regression use kernel function (,)K x y , if the kernel function is turned to be Gaussian function, then the regression function will be1()()(,)ni i i i f x K x v b αα*==-+∑ (11)Wheni i αα*-is not equal to zero, the accordingsamples will be support vectors, we assume thenumber of the support vectors which are gotten after SVM training is g ()g n ≤, support vector are ,1,...,i v i g =, the weight factorsare i i i w αα*=- ,the bias is b,then we use these to construct RBF networks.4 Application and simulation researchWe use RBF networks optimized by SVM insystem identification to test the learning performances and generalization abilities of this algorithm, we also compare it with generalized regression neural network Consuming the identification system denoted by the non-linear discrete function:22(1)()()1(1)y k y k u k y k -=++- (12) To this system, we use cascade- parallel models, the RBF network has two inputs and one output. The twenty samples ((1),(2),...,(20))u u u are selected from [0,2], started from zero, the interval value is 0.1, andtherespondingoutputsare((1),(2),...,(20))y y y ,when the identification isfinished, we use the follow stimulus signal to test the generalization abilities of the networks.(1)Generalized regression neural network select Gaussian function as the kernel function, the distribution density Spread is 0.7, the identification results is shown in figure2 , the mean squares error between identification results and output matrix is 0.0842.And the testing results is shown in figure 3, the mean squares error between testing results and expected output matrix is 0.0282.(2)In optimizing neural networks by SVM, we set the maximum genetic generations is 20, we get the maximum value of fitness by genetic algorithm searching, so we get the model parameters x ＝[379.9678 1.0221 0.0426],that penalty factor C is 379.9678, the width of Gaussian function σ is 1.0221, the insensitive loss function εis 0.0426, furthermore we get the number of support vectors is 4 by the learning of input and output datum, so we set the number of hidden units in RBF network is 4, the()0.15sin(2/25)0.22sin(2/35)0.2,1,2,...,100u k k k k ππ=++=Gaussian function center vector are the support vectors, the width of Gaussian function is thewith the regression machine.toi i iwαα*=-, we know the respondingweights [-1.7610 1.0219 -1.0406 1.7797]w=,and bias value of RBF networks, then the RBFconstructed, just as shown in figure 4.Accordingidentification results shown in figure 5, thematrix is 0.0036.And the testing results is shownfigure 6, the mean squares error betweenresults and expected output matrix is 0.0134.In terms of identification accuracy andgeneralization abilities, the simulation results showthat RBF networks optimized by SVM algorithm issuperior to generalized regression neural network, alsoneedn’t pre-specify the structure by a lot ofexperiments.Figure 2 The identification results of generalized regressionneural networkFigure3 the testing results of generalized regression neuralnetworkFigure4 RBF network structure based on SVMFigure5 the identification results of RBF optimized by SVMFigure6 the testing results of RBF optimized by SVMConclusionThe algorithm we proposed in this paper first usegenetic algorithm to optimize SVM model parameters,then to make certain the structure and parameters ofRBF networks by SVM, which is applied in system identification. The simulation results show this algorithm is more efficient in learning and training, with good identification accuracy and generalization capacity in term of system identificationReferences[1] PETER Andras. The equivalence of support vector machineandregularization neural networks [ J ] . Neural Processing Letters ,2002 , 15 (2) : 97 - 104[2]Yuan Xiao-fang, Wang Yaonan, Sun Wei, Yang Huiqian. A Hybrid Learning Algorithm for RBF Neural Network Based on Support Vector Machines and BP Algorithm[J].Journal of Hunan university,2005,32(3):88-92.[3]Zhang Gup-yun. Support Vector Machine and its application Research[D],Hunan: Hunan University,2006.[4] Zhu Ming-xing,Zhang De-long. Study on the Algorithms of Selecting the Radial Basis Function Cente[J].Journal of Anhui university,2003,24(1):72~78[5] Yang Xu,Ji Y u-bo,Tian Xue.Parameters Selection of SVM Based on Genetic Algorithm[J]Journal of Liaoning University of Petroleum ＆Chemical Technology.，2004，24(1)：54～58[6] Zheng Chun-hong,Jiao Li-cheng,Ding Ai-ling.Automatic model selection for support vector machines using heuristic genetic algorithm [J].Control Theory & Applications，2006，23（2）：187～192[7]Chai Jie,Jiang Qing-yin,Cao Zhi-kai.FunctionApproximation Capability and Algorithms of RBF Neural Net Works[J].Pattern Recognition and Artificial Intelligence，2002，15(3)：310～316[8]Shan Xiao-hong. RBF Neural Networks based on Genetic Algorithm and its Applications in System identification[D].Shandong:Qingdao University,2006.。
