A+Data+Preprocessing+Algorithm+Based-on+SVM+in+Data+Warehouse
- 格式：pdf
- 大小：122.50 KB
- 文档页数：4
A Data Preprocessing Algorithm Based-on SVM in Data Warehouse
Wangjianfen1，Shichanghong2
1School of Information and Electronic Engineering, Zhejiang University of Science and Technology, China, 310023 2 The Quartermaster Institute of the General Logistics Department of the P.L.A, China, 100010
the best generalization ability. SVM has more cases applied in classification. Through the study, SVM can automatically find the support vectors that have good distinguish ability for classification. The classifier constructed by this may maximize the interval between the classes, and has good generalization performance and higher accuracy rate. SVM has already been used to isolate handwriting recognition and speech recognition. But for web page, a kind of semi-structured data, large-scale datasets in data warehouse, SVM training time is too long because of more training examples, therefore unacceptable.
Using SVM multi-categories classifier, each kind of recognition is regarded as an independent two-category problem. Assume that all web pages be divided into k categories, such as L = {a1, a2... ak), set the number of web pages which belongs to category ai is Ni. So for each category ai, the total of training positive example is Ni, then the total of training
The basic SVM method classifies all data into two categories. The method must be extended for N-category classified problem. The simplest method is separate the i category from others. Thus Ncategories classified problem is transformed twocategories classified problem. This method must create N sub-svm classifier based two-catalogues for n-categories stylebooks, and each sub-svm classified one catalogues of the n-catalogues. Obviously the separate super –plane of the method has better classify and generalize ability, because the stylebook set of each sub-svm includes all train stylebooks. But the shortcomings are also the bigger train set and difficult to train, and all sub-svm must be train again in case the new categories added.
A. SVM -decision-tree method
SVM-decision-tree method combines SVM and binary decision tree to form multi-categories classifier. The corresponding relationship exists between N-1 category classification problem (N >2) and two-category classification problem. If a classification problem can be classified by N-categories, then any two kinds of N-categories could be classified, vice verse. Because svm is two- categories classifier, it is natural to combine the svm method with binary-decision tree method. We can obtain a new classifier for a large number of categories. The method is named SVM-decision- tree.The decision tree method should construct several sub-SVM classifiers. The number of sub-SVM classifiers is N-1. There are various plans for constructing a strict binary tree with n-leaves nodes.
Abstract: As real-world data tends to be incomplete, noisy and inconsistent, data preprocessing is an important issue for both data warehouse and data mining. Besides well-structured data, data warehouse integrates semi-structured data from WWW data source and those exterior file data without structure. This paper presents a preprocessing classification algorithm that is based on SVM-decision tree. The multiple-categories classifier is composed of SVM and binary decision tree and used for data classification in data warehouse. It can reduce the train scale of SVM classifier and improve the training efficiency. The experiment that classify Chinese Web Page, one kinds of semi-structured data, with this algorithm shows that it not only reduces the size of train set but also has very high training efficiency. Its precision and recall are also very good. Key words: SVM;data preprocessing; SVM-decision tree；data warehouse
648
II. A Classify algorithm Based on SVM-decision Tree
Besides well-structured data, data warehouses integrate semi-structured data from WWW data source and those exterior file data without structure. These data sources have not only different data models but also different query ability. Chinese Web Page is a kind of semi-structured data from WWW data source. We adopt svm-decision-tree methods classified the Chinese web page to test its performances.
SVM method is established based on VC dimension theory and minimum structure risk principle of statistic study theory. It seeks finest compromise between model complexity and study ability according to limited stylebook information and obtain
Wangjianfen1，Shichanghong2
1School of Information and Electronic Engineering, Zhejiang University of Science and Technology, China, 310023 2 The Quartermaster Institute of the General Logistics Department of the P.L.A, China, 100010
the best generalization ability. SVM has more cases applied in classification. Through the study, SVM can automatically find the support vectors that have good distinguish ability for classification. The classifier constructed by this may maximize the interval between the classes, and has good generalization performance and higher accuracy rate. SVM has already been used to isolate handwriting recognition and speech recognition. But for web page, a kind of semi-structured data, large-scale datasets in data warehouse, SVM training time is too long because of more training examples, therefore unacceptable.
Using SVM multi-categories classifier, each kind of recognition is regarded as an independent two-category problem. Assume that all web pages be divided into k categories, such as L = {a1, a2... ak), set the number of web pages which belongs to category ai is Ni. So for each category ai, the total of training positive example is Ni, then the total of training
The basic SVM method classifies all data into two categories. The method must be extended for N-category classified problem. The simplest method is separate the i category from others. Thus Ncategories classified problem is transformed twocategories classified problem. This method must create N sub-svm classifier based two-catalogues for n-categories stylebooks, and each sub-svm classified one catalogues of the n-catalogues. Obviously the separate super –plane of the method has better classify and generalize ability, because the stylebook set of each sub-svm includes all train stylebooks. But the shortcomings are also the bigger train set and difficult to train, and all sub-svm must be train again in case the new categories added.
A. SVM -decision-tree method
SVM-decision-tree method combines SVM and binary decision tree to form multi-categories classifier. The corresponding relationship exists between N-1 category classification problem (N >2) and two-category classification problem. If a classification problem can be classified by N-categories, then any two kinds of N-categories could be classified, vice verse. Because svm is two- categories classifier, it is natural to combine the svm method with binary-decision tree method. We can obtain a new classifier for a large number of categories. The method is named SVM-decision- tree.The decision tree method should construct several sub-SVM classifiers. The number of sub-SVM classifiers is N-1. There are various plans for constructing a strict binary tree with n-leaves nodes.
Abstract: As real-world data tends to be incomplete, noisy and inconsistent, data preprocessing is an important issue for both data warehouse and data mining. Besides well-structured data, data warehouse integrates semi-structured data from WWW data source and those exterior file data without structure. This paper presents a preprocessing classification algorithm that is based on SVM-decision tree. The multiple-categories classifier is composed of SVM and binary decision tree and used for data classification in data warehouse. It can reduce the train scale of SVM classifier and improve the training efficiency. The experiment that classify Chinese Web Page, one kinds of semi-structured data, with this algorithm shows that it not only reduces the size of train set but also has very high training efficiency. Its precision and recall are also very good. Key words: SVM;data preprocessing; SVM-decision tree；data warehouse
648
II. A Classify algorithm Based on SVM-decision Tree
Besides well-structured data, data warehouses integrate semi-structured data from WWW data source and those exterior file data without structure. These data sources have not only different data models but also different query ability. Chinese Web Page is a kind of semi-structured data from WWW data source. We adopt svm-decision-tree methods classified the Chinese web page to test its performances.
SVM method is established based on VC dimension theory and minimum structure risk principle of statistic study theory. It seeks finest compromise between model complexity and study ability according to limited stylebook information and obtain
