下载文档原格式
数据挖掘的概念与技术介绍数据挖掘的概念与技术介绍数据挖掘是指从大量的数据中发现隐藏在其中的有价值的信息、模式和规律的过程。
随着互联网时代的到来，越来越多的数据被收集和存储，数据挖掘成为了从这些海量数据中获取洞察和知识的重要工具。
本文将围绕数据挖掘的概念和技术展开讨论，帮助读者深入理解数据挖掘的核心要素和方法。
一、数据挖掘的概念1.1 数据挖掘的定义数据挖掘是一种通过自动或半自动的方式，从大量的数据中发现有用的信息、模式和规律的过程。
通过应用统计学、机器学习和人工智能等技术，数据挖掘可以帮助人们从数据中进行预测、分析和决策。
1.2 数据挖掘的目标数据挖掘的主要目标是从数据中发现隐藏的模式和规律，并将这些知识应用于实际问题的解决。
数据挖掘可以帮助企业提高市场营销的效果、改进产品设计、优化生产过程等。
数据挖掘也被广泛应用于科学研究、金融风险分析、医学诊断等领域。
1.3 数据挖掘的流程数据挖掘的流程通常包括数据收集、数据预处理、模型构建、模型评估和模型应用等步骤。
其中，数据预处理是数据挖掘流程中非常重要的一环，它包括数据清洗、数据集成、数据变换和数据规约等子任务。
二、数据挖掘的技术2.1 关联规则挖掘关联规则挖掘是数据挖掘的一个重要技术，它用于发现数据集中的项之间的关联关系。
通过挖掘关联规则，可以发现数据中隐藏的有用信息，如购物篮分析中的“啤酒和尿布”现象。
2.2 分类与回归分类与回归是数据挖掘中常用的技术，它们用于对数据进行分类或预测。
分类是指根据已有的样本数据，建立分类模型，然后将新的数据实例分到不同的类别中。
回归则是根据数据的特征和已知的输出值，建立回归模型，然后预测新的数据实例的输出值。
2.3 聚类分析聚类分析是一种将数据分成不同的类别或簇的技术。
通过发现数据之间的相似性，聚类可以帮助人们理解数据的内在结构和特点。
聚类分析在市场细分、社交网络分析等领域具有广泛的应用。
2.4 异常检测异常检测是指从数据中识别出与大多数数据显著不同的样本或模式。
第一章数据仓库与数据挖掘概述1.数据库与数据仓库的本质差别是什么？答：数据库用于事务处理，数据仓库用于决策分析;数据库保持事务处理的当前状态，数据仓库既保存过去的数据又保存当前的数据;数据仓库的数据是大量数据库的集成;对数据库的操作比较明确，操作数据量少,对数据仓库操作不明确，操作数据量大。
数据库是细节的、在存取时准确的、可更新的、一次操作数据量小、面向应用且支持管理；数据仓库是综合或提炼的、代表过去的数据、不更新、一次操作数据量大、面向分析且支持决策。
6.说明OLTP与OLAP的主要区别。
答：OLTP针对的是细节性数据、当前数据、经常更新、一次性处理的数据量小、对响应时间要求高且面向应用，事务驱动； OLAP针对的是综合性数据、历史数据、不更新，但周期性刷新、一次处理的数据量大、响应时间合理且面向分析，分析驱动。
8.元数据的定义是什么？答：元数据（metadata）定义为关于数据的数据（data about data），即元数据描述了数据仓库的数据和环境。
9.元数据与数据字典的关系什么？答：在数据仓库中引入了“元数据”的概念，它不仅仅是数据仓库的字典，而且还是数据仓库本身信息的数据。
18.说明统计学与数据挖掘的不同。
答：统计学主要是对数量数据（数值）或连续值数据（如年龄、工资等），进行数值计算（如初等运算）的定量分析，得到数量信息。
数据挖掘主要对离散数据（如职称、病症等）进行定性分析（覆盖、归纳等），得到规则知识。
19.说明数据仓库与数据挖掘的区别与联系。
答：数据仓库是一种存储技术，它能适应于不同用户对不同决策需要提供所需的数据和信；数据挖掘研究各种方法和技术，从大量的数据中挖掘出有用的信息和知识。
数据仓库与数据挖掘都是决策支持新技术。
但它们有着完全不同的辅助决策方式。
在数据仓库系统的前端的分析工具中，数据挖掘是其中重要工具之一。
它可以帮助决策用户挖掘数据仓库的数据中隐含的规律性。
数据仓库和数据挖掘的结合对支持决策会起更大的作用。
数据仓库与数据挖掘第二版课程设计一、课程简介本课程旨在为学生介绍数据仓库与数据挖掘的基本概念和技术。
本课程将从数据仓库的基本概念、设计和建模、ETL技术、数据仓库应用及数据挖掘技术等多个方面进行探讨。
通过本课程的学习，学生将掌握数据仓库和数据挖掘的基本技术和应用，培养数据分析和挖掘能力，为其未来的工作和学习提供基础。
二、课程目标本课程主要目的是：1.介绍数据仓库和数据挖掘的概念和技术，了解数据仓库的设计和建模方法，掌握数据仓库的应用和数据挖掘的基本算法；2.能够使用商业化数据仓库和数据挖掘工具进行基本的数据分析和挖掘实践；3.能够基于真实数据，进行数据仓库设计和建模，并进行相应的数据挖掘分析；4.能够独立思考和解决具体数据仓库和数据挖掘问题；三、课程安排第一周知识点•数据仓库基本概念•数据仓库架构设计实践•数据仓库案例分析知识点•数据仓库建模方法•算法基础实践•数据仓库设计第三周知识点•数据仓库ETL技术•数据仓库性能调优实践•数据仓库ETL设计第四周知识点•数据仓库应用•数据挖掘基本算法实践•数据挖掘案例分析四、课程评估期末项目设计在期末作业中，学生需要设计一个基于真实数据的数据仓库，并使用相应的数据挖掘技术进行分析。
期末项目设计占总成绩的60%。
平时作业和课堂表现占总成绩的40%。
平时作业主要包括练习题、小型数据仓库设计等。
五、参考资料1.Ralph Kimball等编著，《数据仓库工具包》；2.邵斌等编著，《数据挖掘导论》；3.刘元春等编著，《数据仓库》；4.吴军，《数学之美》。
数据挖掘导论第二版课程设计一、引言随着信息技术和互联网的快速发展，数据量呈现爆炸式增长，数据挖掘越来越成为一项热门领域。
本次课程设计旨在让学生了解数据挖掘的基本概念、主要任务、数据挖掘流程和常用算法，培养学生的数据挖掘技能和实践能力。
二、课程设计方案1. 设计目标本次课程设计的主要目标是让学生掌握以下能力：•理解数据挖掘的基本概念和任务•掌握数据挖掘的主要流程和方法•熟悉常用的数据挖掘算法及其应用•能够运用数据挖掘工具进行数据分析和挖掘2. 设计内容本次课程设计主要包括以下内容：（1）数据挖掘基础•数据挖掘的定义和应用•数据挖掘任务•数据挖掘流程（2）数据探索与预处理•数据清洗•数据集成•数据变换•数据规约（3）监督学习算法•决策树•朴素贝叶斯•K近邻•支持向量机（4）无监督学习算法•聚类分析•关联规则挖掘•主成分分析（5）数据挖掘工具使用•Python数据挖掘工具：Scikit-Learn和Pandas•R语言数据挖掘工具：Rattle和RapidMiner（6）实践案例•基于监督学习算法的图书馆借阅预测•基于无监督学习算法的客户价值分析3. 设计方法本次课程设计采用以下教学方法：•理论授课：介绍数据挖掘基础、算法原理和工具使用方法•实操演练：使用Python和R工具进行实践操作和案例讲解•课堂讨论：学生分组讨论数据挖掘实践案例和挑战题目•评估测试：布置数据挖掘任务并进行实践评估和算法选择测试4. 设计评估为了评估学生的学习成果和教学效果，本次课程设计采用以下评估方法：•课堂表现：包括学生的积极性、参与度、提问和回答精度等•作业成绩：包括课后作业和实践操作任务成绩•期末项目：学生需要完成一个选题的数据挖掘项目，并进行展示和答辩三、结语通过本次课程设计，学生将掌握数据挖掘的基本概念、主要流程和常用算法等技能，并能够熟练使用数据挖掘工具进行数据分析和挖掘。
这不仅对学生的个人发展非常有帮助，也有助于提高社会掌握数据挖掘技术的能力，更好地应对信息社会的挑战。
数据挖掘与数据仓库（教学大纲）Data mining and data warehouse课程编码：05405140 学分： 2.5 课程类别： 专业方向课计划学时： 48 其中讲课：32 实验或实践： 上机：16适用专业：信息管理与信息系统、电子商务推荐教材：陈文伟，数据仓库与数据挖掘教程，清华大学出版社，2008参考书目：1. Richard J. Roiger, Michael W. Geatz. Data Mining: A Tutorial-Based Primer.2003.2. Ian H. Witten, Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques(第二版). 机械工业出版社（影印版），2005.3. Jiawei Han, Micheline Kamber. Data Mining: Concepts and Techniques.2001.5.4. 数据仓库与数据挖掘技术（第2版），陈京民 编著，电子工业出版社，2007.115. 数据仓库和数据挖掘，苏新宁 等编著，清华大学出版社，2006.46. 数据挖掘Clementine应用实务，谢邦昌 主编，机械工业出版社，2008.4课程的教学目的与任务本课程将系统介绍数据挖掘的基本概念、基本原理和应用基础，通过课堂讲授、实例分析，提高学生数据挖掘技术的认识，熟悉基本工具应用，并掌握设计和开发数据挖掘算法和系统的初步能力。
课程的基本要求1、了解数据仓库及数据挖掘的概念、特征、应用范围，以及主要数据挖掘工具2、了解OLTP 和 OLAP的区别；熟悉OLAP 的体系结构，以及如何评价OLAP工具；掌握多维分析的基本分析动作。
3、了解数据质量，掌握数据预处理方法，4、掌握数据挖掘的定性归纳技术、关联挖掘、聚类分析、分类方法、预测方法、文本挖掘、WEB挖掘5、熟练掌握数据挖掘软件Clementine在各类挖掘任务中的应用。
464IEEETRANSACTIONSONNEURALNETWORKS,VOL.14,NO.2,MARCH2003BookReview_____________________________________________________________________________
DataMining:Concepts,Models,MethodsandAlgorithmsMehmoodKantardzic(NewYork:JohnWileyandIEEE,2003,ISBN0-471-22852-4.ReviewedbyJacekM.ZuradaTheimportanceofdataminingarisesfromthefactthatthemodernworldisincreasinglydata-driven.Wearesurroundedbydatainnumer-ical,symbolic,verbalandvisualformats,tonameafew.Thedata,oftenraw,mustbeanalyzedandprocessedtoconvertitintootherformatsthatinform,instruct,answer,orotherwiseaidunderstandingandassistusindecision-making.IntheageofInternet,intranets,datawarehousesanddatamarts,availabledatasetshavegrowninsizeandcomplexity.This,inturn,hasleadtoaninevitableshiftawayfromdirecthands-ondataanalysistowardindirect,automaticandintelligentdataanalysisinwhichtheanalystworksviaincreasinglycomplexandsophisticatedsoftwaretools.Thisgrowthhasalsobroughttheemergenceofanewareaofin-terestindatamining,developedespeciallytoextractvaluableinforma-tionfromverylargedatasets.Shortofbeinganewdisciplineequippedwithanewcoreofknowledge,dataminingoffersaninterdisciplinarycollectionofspecializedtechniquesthatallowforintegrationanden-hancementofresultsfromdisciplinessuchasstatistics,artificialintel-ligence,databases,patternrecognition,andcomputervisualization.Thisnewbookprovidesasystematicreviewofstate-of-the-artmethodologiesandtechniquesforanalyzinglargequantitiesofrawdatainhigh-dimensionaldataspaces,andforextractionofnewinfor-mationfordecision-making.Whiledatamininghasitsrootsinseveraldisciplines,mostexistingbookshavebeenbiasedwithbackgroundandemphasisinoneofthefieldssuchasdatabases,statistics,patternrecognition,artificialintelligence,orbusinessinformationsystems[1]–[8].Thisbookistryingtopresentacomprehensiveandbalancedapproachbyincludingmostoftherelevantresultsindatabases,ma-chinelearning,statistics,computervisualization,andsoftcomputingthatarecontributingtothedataminingfield.ThescopeofthebookissomewhatsimilartoBerthold’sandHand’svolume[9],yetthelevelofpresentationmakesitappealingtothereadersthathavelessrigorousmathematicalbackground.Thebookcoversawiderangeofcurrenttheoreticalandpracticalissues.Theemphasisofthepresentationisondetailedexplanationsofrelevantphasesandtechniquesofthedataminingprocess.Thesearefollowedbyillustrative,oftennumerical,examples.Thelucentapproachislikelytowidenthereadershipforthisbookascomparedwithmanyotherbooksthatrequirestrongermathematicsorcomputerscienceprerequisites.Althoughthebookisintroductoryinnature,elementarybackgroundinalgorithms,statistics,datastructures,anddatabaseswillbehelpful.AfterintroducingintroductoryconceptsofdatamininginChapter1,Chapters2–3covercommoncharacteristicsofrawlargedatasetsandtypicaltechniquesofdatapreprocessing.Topicscoveredincludetrans-formations,handlingofmissingdataandoutliers,reductionoffea-turesanddatasets,entropy,andprincipalcomponentanalysis,tonameafew.ThetextemphasizestheimportanceoftheseinitialphasesofDigitalObjectIdentifier10.1109/TNN.2003.809639transformingandreducinglargerawdatasetsforthefinalsuccessandqualityofdataminingresults.Chapter4introducesthetheoreticalframeworkoflearninginob-servationalenvironment.Chapters5–11provideanoverviewofcom-monlyusedapproachesofdataminingtechniquessuchasstatisticalin-ference,clustering,logic-basedtechniques,associationrules,webandtextminingtechniques,artificialneuralnetworks,geneticalgorithms,andfuzzysystems.Chapter12reviewsanumberofvisualizationtech-niquesespeciallyusefulforrepresentationofhigh-dimensionalsam-ples.Appendixesprovideanoverviewofcommerciallyandpubliclyavailabledataminingpackagesandofferanextensiveaddresslistofrelevantwebsitesandsoftwarevendors.Bytimelypublishingofthisbook,Dr.Kantardzichasprovidedthetechnicalcommunitywithaninformativeandveryreadablevolumeinthisfastgrowingarea.Thevolumeisalsoaverygoodcandidateforatextbookinanintroductorydataminingcourse.Notonlyisitorganizedinasystematic,pedagogicalway,butalsoprovidesanumberofReviewQuestionsandProblemsattheendofeachchapter.Theseshouldbeveryhelpfulforadoptersandstudentsalike.Infact,thisbookmayencourageinstructorstodevelopanewcourseindataminingattheupperundergraduateorattheentrygraduatelevelinengineeringandcomputerscienceprograms.
About the T utorialData Mining is defined as the procedure of extracting information from huge sets of data. In other words, we can say that data mining is mining knowledge from data.The tutorial starts off with a basic overview and the terminologies involved in data mining and then gradually moves on to cover topics such as knowledge discovery, query language, classification and prediction, decision tree induction, cluster analysis, and how to mine the Web.AudienceThis tutorial has been prepared for computer science graduates to help them understand the basic-to-advanced concepts related to data mining.PrerequisitesBefore proceeding with this tutorial, you should have an understanding of the basic database concepts such as schema, ER model, Structured Query language and a basic knowledge of Data Warehousing concepts.Copyright & DisclaimerCopyright 2014 by Tutorials Point (I) Pvt. Ltd.All the content and graphics published in this e-book are the property of Tutorials Point (I) Pvt. Ltd. The user of this e-book is prohibited to reuse, retain, copy, distribute or republish any contents or a part of contents of this e-book in any manner without written consent of the publisher.We strive to update the contents of our website and tutorials as timely and as precisely as possible, however, the contents may contain inaccuracies or errors. Tutorials Point (I) Pvt. Ltd. provides no guarantee regarding the accuracy, timeliness or completeness of our website or its contents including this tutorial. If you discover any errors on our website or inthistutorial,******************************************T able of ContentsAbout the Tutorial (i)Audience (i)Prerequisites (i)Copyright & Disclaimer (i)Table of Contents .................................................................................................................................... i i 1.DATA MINING ─ OVERVI EW . (1)What is Data Mining? (1)Data Mining Applications (1)Market Analysis and Management (1)Corporate Analysis and Risk Management (2)Fraud Detection (2)2.DATA MINING ─ TASKS (3)Descriptive Function (3)Classification and Prediction (4)Data Mining Task Primitives (5)3.DATA MINING ─ ISSUES (7)Mining Methodology and User Interaction Issues (7)Performance Issues (8)Diverse Data Types Issues (8)4.DATA MINING ─ EVALUA TION (9)Data Warehouse (9)Data Warehousing (9)Query-Driven Approach (9)Update-Driven Approach (10)From Data Warehousing (OLAP) to Data Mining (OLAM) (10)Importance of OLAM (11)5.DATA MINING ─ TERMIN OLOGIES (13)Data Mining (13)Data Mining Engine (13)Knowledge Base (13)Knowledge Discovery (13)User Interface (14)Data Integration (14)Data Cleaning (14)Data Selection (14)Clusters (14)Data Transformation (15)6.DATA MINING ─ KNOWLE DGE DISCOVERY (16)What is Knowledge Discovery? (16)7.DATA MINING ─ SYSTEM S (17)Data Mining System Classification (17)Integrating a Data Mining System with a DB/DW System (19)8.DATA MINING ─ QUERY LANGUAGE (20)Syntax for Task-Relevant Data Specification (20)Syntax for Specifying the Kind of Knowledge (20)Syntax for Concept Hierarchy Specification (22)Syntax for Interestingness Measures Specification (23)Syntax for Pattern Presentation and Visualization Specification (23)Full Specification of DMQL (23)Data Mining Languages Standardization (24)9.DATA MINING ─ CLASSI FICATION AND PREDICTION (25)What is Classification? (25)What is Prediction? (25)How Does Classification Work? (25)Classification and Prediction Issues (27)Comparison of Classification and Prediction Methods (27)10.DATA MINING ─ DECISI ON TREE INDUCTION (28)Decision Tree Induction Algorithm (28)Tree Pruning (30)Cost Complexity (30)11.DATA MINING ─ BAYESI AN CLASSIFICATION (31)Bayes' Theorem (31)Bayesian Belief Network (31)Directed Acyclic Graph (31)Directed Acyclic Graph Representation (32)Conditional Probability Table (32)12.DATA MINING ─ RULE-BASED CLASSIFICATION (33)IF-THEN Rules (33)Rule Extraction (33)Rule Induction Using Sequential Covering Algorithm (33)Rule Pruning (34)13.DATA MINING ─ CLASSI FICATION METHODS (35)Genetic Algorithms (35)Rough Set Approach (35)Fuzzy Set Approach (36)14.DATA MINING ─ CLUSTE R ANALYSIS (38)What is Clustering? (38)Applications of Cluster Analysis (38)Requirements of Clustering in Data Mining (38)Clustering Methods (39)15.DATA MINING ─ MINING TEXT DATA (42)Information Retrieval (42)Basic Measures for Text Retrieval (42)16.DATA MINING ─ MINING WORLD WIDE WEB (44)Challenges in Web Mining (44)Mining Web Page Layout Structure (44)Vision-based Page Segmentation (VIPS) (44)17.DATA MINING ─ APPLIC ATIONS AND TRENDS (46)Data Mining Applications (46)Data Mining System Products (48)Choosing a Data Mining System (48)Trends in Data Mining (49)18.DATA MINING ─ THEMES (51)Theoretical Foundations of Data Mining (51)Statistical Data Mining (52)Visual Data Mining (53)Audio Data Mining (53)Data Mining and Collaborative Filtering (54)Data Mining 6There is a huge amount of data available in the Information Industry. This data is of no use until it is converted into useful information. It is necessary to analyze this huge amount of data and extract useful information from it.Extraction of information is not the only process we need to perform; data mining also involves other processes such as Data Cleaning, Data Integration, Data Transformation, Data Mining, Pattern Evaluation and Data Presentation. Once all these processes are over, we would be able to use this information in many applications such as Fraud Detection, Market Analysis, Production Control, Science Exploration, etc.What is Data Mining?Data Mining is defined as extracting information from huge sets of data. In other words, we can say that data mining is the procedure of mining knowledge from data. The information or knowledge extracted so can be used for any of the following applications: ∙Market Analysis ∙Fraud Detection ∙Customer Retention ∙Production Control ∙ Science ExplorationData Mining ApplicationsData mining is highly useful in the following domains:∙Market Analysis and Management ∙Corporate Analysis & Risk Management ∙ Fraud DetectionApart from these, data mining can also be used in the areas of production control, customer retention, science exploration, sports, astrology, and Internet Web Surf-Aid Market Analysis and ManagementListed below are the various fields of market where data mining is used:∙ Customer Profiling - Data mining helps determine what kind of people buy what kind of products.1.Data Mining7∙Identifying Customer Requirements- Data mining helps in identifying the best products for different customers. It uses prediction to find the factors that may attract new customers.∙Cross Market Analysis- Data mining performs Association/correlations between product sales.∙Target Marketing- Data mining helps to find clusters of model customers who share the same characteristics such as interests, spending habits, income, etc.∙Determining Customer purchasing pattern- Data mining helps in determining customer purchasing pattern.∙Providing Summary Information- Data mining provides us various multidimensional summary reports.Corporate Analysis and Risk ManagementData mining is used in the following fields of the Corporate Sector:∙Finance Planning and Asset Evaluation- It involves cash flow analysis and prediction, contingent claim analysis to evaluate assets.∙Resource Planning- It involves summarizing and comparing the resources and spending.∙Competition- It involves monitoring competitors and market directions.Fraud DetectionData mining is also used in the fields of credit card services and telecommunication to detect frauds. In fraud telephone calls, it helps to find the destination of the call, duration of the call, time of the day or week, etc. It also analyzes the patterns that deviate from expected norms.Data Mining 8Data mining deals with the kind of patterns that can be mined. On the basis of the kind of data to be mined, there are two categories of functions involved in Data Mining: ∙Descriptive ∙ Classification and PredictionDescriptive FunctionThe descriptive function deals with the general properties of data in the database. Here is the list of descriptive functions:∙Class/Concept Description ∙Mining of Frequent Patterns ∙Mining of Associations ∙Mining of Correlations ∙ Mining of ClustersClass/Concept DescriptionClass/Concept refers to the data to be associated with the classes or concepts. For example, in a company, the classes of items for sales include computer and printers, and concepts of customers include big spenders and budget spenders. Such descriptions of a class or a concept are called class/concept descriptions. These descriptions can be derived by the following two ways:∙Data Characterization - This refers to summarizing data of a class under study. This class under study is called as the Target Class. ∙ Data Discrimination - It refers to the mapping or classification of a class with some predefined group or class.Mining of Frequent PatternsFrequent patterns are those patterns that occur frequently in transactional data. Here is the list of kind of frequent patterns:∙Frequent Item Set - It refers to a set of items that frequently appear together, for example, milk and bread. ∙ Frequent Subsequence - A sequence of patterns that occur frequently such as purchasing a camera is followed by memory card.2.Data Mining9∙Frequent Sub Structure- Substructure refers to different structural forms, such as graphs, trees, or lattices, which may be combined with item-sets or subsequences.Mining of AssociationAssociations are used in retail sales to identify patterns that are frequently purchased together. This process refers to the process of uncovering the relationship among data and determining association rules.For example, a retailer generates an association rule that shows that 70% of time milk is sold with bread and only 30% of times biscuits are sold with bread.Mining of CorrelationsIt is a kind of additional analysis performed to uncover interesting statistical correlations between associated-attribute-value pairs or between two item sets to analyze that if they have positive, negative or no effect on each other.Mining of ClustersCluster refers to a group of similar kind of objects. Cluster analysis refers to forming group of objects that are very similar to each other but are highly different from the objects in other clusters.Classification and PredictionClassification is the process of finding a model that describes the data classes or concepts. The purpose is to be able to use this model to predict the class of objects whose class label is unknown. This derived model is based on the analysis of sets of training data. The derived model can be presented in the following forms:∙Classification (IF-THEN) Rules∙Decision Trees∙Mathematical Formulae∙Neural NetworksThe list of functions involved in these processes are as follows:∙Classification- It predicts the class of objects whose class label is unknown. Its objective is to find a derived model that describes and distinguishes data classes or concepts. The Derived Model is based on the analysis set of training data i.e. the data object whose class label is well known.∙Prediction- It is used to predict missing or unavailable numerical data values rather than class labels. Regression Analysis is generally used for prediction. Prediction can also be used for identification of distribution trends based on available data.10∙Outlier Analysis- Outliers may be defined as the data objects that do not comply with the general behavior or model of the data available.∙Evolution Analysis- Evolution analysis refers to the description and model regularities or trends for objects whose behavior changes over time.Data Mining T ask Primitives∙We can specify a data mining task in the form of a data mining query.∙This query is input to the system.∙ A data mining query is defined in terms of data mining task primitives.Note: These primitives allow us to communicate in an interactive manner with the data mining system. Here is the list of Data Mining Task Primitives:∙Set of task relevant data to be mined.∙Kind of knowledge to be mined.∙Background knowledge to be used in discovery process.∙Interestingness measures and thresholds for pattern evaluation.∙Representation for visualizing the discovered patterns.Set of task relevant data to be minedThis is the portion of database in which the user is interested. This portion includes the following:∙Database Attributes∙Data Warehouse dimensions of interestKind of knowledge to be minedIt refers to the kind of functions to be performed. These functions are:∙Characterization∙Discrimination∙Association and Correlation Analysis∙Classification∙Prediction∙Clustering∙Outlier Analysis11∙Evolution AnalysisBackground knowledgeThe background knowledge allows data to be mined at multiple levels of abstraction. For example, the Concept hierarchies are one of the background knowledge that allows data to be mined at multiple levels of abstraction.Interestingness measures and thresholds for pattern evaluationThis is used to evaluate the patterns that are discovered by the process of knowledge discovery. There are different interesting measures for different kind of knowledge.Representation for visualizing the discovered patternsThis refers to the form in which discovered patterns are to be displayed. These representations may include the following:∙Rules∙Tables∙Charts∙Graphs∙Decision Trees∙CubesData Mining12Data mining is not an easy task, as the algorithms used can get very complex and data is not always available at one place. It needs to be integrated from various heterogeneous data sources. These factors also create some issues. Here in this tutorial, we will discuss the major issues regarding:∙ Mining Methodology and User Interaction ∙ Performance Issues ∙Diverse Data Types IssuesThe following diagram describes the major issues.Mining Methodology and User Interaction IssuesIt refers to the following kinds of issues:∙Mining different kinds of knowledge in databases - Different users may be interested in different kinds of knowledge. Therefore it is necessary for data mining to cover a broad range of knowledge discovery task.3. DATA MINING ─ ISSUESData Mining13∙Interactive mining of knowledge at multiple levels of abstraction- The data mining process needs to be interactive because it allows users to focus the search for patterns, providing and refining data mining requests based on the returned results.∙Incorporation of background knowledge- To guide discovery process and to express the discovered patterns, the background knowledge can be used. Background knowledge may be used to express the discovered patterns not only in concise terms but at multiple levels of abstraction.∙Data mining query languages and ad hoc data mining- Data Mining Query language that allows the user to describe ad hoc mining tasks, should be integrated with a data warehouse query language and optimized for efficient and flexible data mining.∙Presentation and visualization of data mining results- Once the patterns are discovered it needs to be expressed in high level languages, and visual representations. These representations should be easily understandable.∙Handling noisy or incomplete data- The data cleaning methods are required to handle the noise and incomplete objects while mining the data regularities. If the data cleaning methods are not there then the accuracy of the discovered patterns will be poor.∙Pattern evaluation- The patterns discovered should be interesting because either they represent common knowledge or lack novelty.Performance IssuesThere can be performance-related issues such as follows:∙Efficiency and scalability of data mining algorithms- In order to effectively extract the information from huge amount of data in databases, data mining algorithm must be efficient and scalable.∙Parallel, distributed, and incremental mining algorithms- The factors such as huge size of databases, wide distribution of data, and complexity of data mining methods motivate the development of parallel and distributed data mining algorithms.These algorithms divide the data into partitions which is further processed in a parallel fashion. Then the results from the partitions is merged. The incremental algorithms, update databases without mining the data again from scratch.Diverse Data T ypes Issues∙Handling of relational and complex types of data- The database may contain complex data objects, multimedia data objects, spatial data, temporal data etc. It is not possible for one system to mine all these kind of data.∙Mining information from heterogeneous databases and global information systems- The data is available at different data sources on LAN or WAN. These data source may be structured, semi structured or unstructured. Therefore mining the knowledge from them adds challenges to data mining.Data Mining14Data Mining15Data WarehouseA data warehouse exhibits the following characteristics to support the management's decision-making process:∙Subject Oriented - Data warehouse is subject oriented because it provides us the information around a subject rather than the organization's ongoing operations. These subjects can be product, customers, suppliers, sales, revenue, etc. The data warehouse does not focus on the ongoing operations, rather it focuses on modelling and analysis of data for decision-making.∙Integrated - Data warehouse is constructed by integration of data from heterogeneous sources such as relational databases, flat files etc. This integration enhances the effective analysis of data.∙Time Variant - The data collected in a data warehouse is identified with a particular time period. The data in a data warehouse provides information from a historical point of view.∙Non-volatile - Nonvolatile means the previous data is not removed when new data is added to it. The data warehouse is kept separate from the operational database therefore frequent changes in operational database is not reflected in the data warehouse.Data WarehousingData warehousing is the process of constructing and using the data warehouse. A data warehouse is constructed by integrating the data from multiple heterogeneous sources. It supports analytical reporting, structured and/or ad hoc queries, and decision making. Data warehousing involves data cleaning, data integration, and data consolidations. To integrate heterogeneous databases, we have the following two approaches:∙ Query Driven Approach ∙Update Driven ApproachQuery-Driven ApproachThis is the traditional approach to integrate heterogeneous databases. This approach is used to build wrappers and integrators on top of multiple heterogeneous databases. These integrators are also known as mediators.4.16Process of Query Driven Approach1.When a query is issued to a client side, a metadata dictionary translates the query intothe queries, appropriate for the individual heterogeneous site involved.2.Now these queries are mapped and sent to the local query processor.3.The results from heterogeneous sites are integrated into a global answer set.DisadvantagesThis approach has the following disadvantages:∙The Query Driven Approach needs complex integration and filtering processes.∙It is very inefficient and very expensive for frequent queries.∙This approach is expensive for queries that require aggregations.Update-Driven ApproachToday's data warehouse systems follow update-driven approach rather than the traditional approach discussed earlier. In the update-driven approach, the information from multiple heterogeneous sources is integrated in advance and stored in a warehouse. This information is available for direct querying and analysis.AdvantagesThis approach has the following advantages:∙This approach provides high performance.∙The data can be copied, processed, integrated, annotated, summarized and restructured in the semantic data store in advance.Query processing does not require interface with the processing at local sources.From Data Warehousing (OLAP) to Data Mining (OLAM)Online Analytical Mining integrates with Online Analytical Processing with data mining and mining knowledge in multidimensional databases. Here is the diagram that shows the integration of both OLAP and OLAM:17Importance of OLAMOLAM is important for the following reasons:∙High quality of data in data warehouses- The data mining tools are required to work on integrated, consistent, and cleaned data. These steps are very costly in the preprocessing of data. The data warehouses constructed by such preprocessing are valuable sources of high quality data for OLAP and data mining as well.∙Available information processing infrastructure surrounding data warehouses- Information processing infrastructure refers to accessing, integration, consolidation, and transformation of multiple heterogeneous databases, web-accessing and service facilities, reporting and OLAP analysis tools.18∙OLAP-based exploratory data analysis- Exploratory data analysis is required for effective data mining. OLAM provides facility for data mining on various subset of data and at different levels of abstraction.∙Online selection of data mining functions- Integrating OLAP with multiple data mining functions and online analytical mining provide users with the flexibility to select desired data mining functions and swap data mining tasks dynamically.19End of ebook previewIf you liked what you saw…Buy it from our store @ https://。
