Agenda

Software Systems Engineering Institute for Computer Science and Business Information Systems (ICB)University of Duisburg-Essen, Germanywww.sse.uni-due.deQualitätssicherung in derSoftware-ProduktlinienentwicklungHerausforderungen und LösungenAndreas MetzgerSoftware Systems EngineeringOverviewResearch Group–Location:-University of Duisburg-Essen-Campus Essen-Germany–Head:Prof. Dr. Klaus Pohl–Team:-15 researchers-15 student employees-Several open positions(Ph.D. and PostDoc)Research Topics–Requirements Engineering–Engineering of Service-based Systems(incl. SOA & QoS)–Software Product Line Engineering(incl. Variability Modeling & Management)–Requirements-based TestingAgendaSoftware Product Line Engineering (SPLE)Quality Assurance in SPLEStrategies and Techniques for Quality Assurance in Domain EngineeringConclusion and PerspectivesMotivationdiversity of products?S o lu t i o n : S o f t w a r e P r o d u c t L i n e E n g i n e e r i n g (S P L E )S o lu t i o n : S o f t w a r e P r o d u c t L i n e E n g i n e e r i n g (S P L E )Key ConceptsPro-active reuse of development artifacts –Requirements, components,code, test cases, documentation, etc.Pro-active reuse in SPLE is……Strategic & planned-Driven by product management-Considering business andtechnology strategy…Systematic-Development process is tailored for reuse-Supported by specific development principles,techniques, and toolsTwo Key Differences……compared with the development of single systems: (1)Two interacting development processes[Pohl et al. 2005]–Domain Engineering (“development for reuse”)–Application Engineering(“development with reuse”)(2) Explicit definition & management of PL variabilityExample:Mobile phones have a lot in commonBut they can vary in terms of-Display sizes,-Functions,-Communication protocols, etc.Documenting Product Line VariabilityVariation PointVariability ConstraintTwo Development ProcessesDomain engineeringApplicationEngineeringApplicationArtifactsDerive individualized applications/productsextending/changing/configuring the reusable artifactsDefine Product Line VariabilityMediaPlayer«component»:UserInterfacePlayMedia«component»:PDFViewerResolve Product Line VariabilityMediaPlayer«component»:UserInterfacePlayMedia«component»:PDFViewerAgendaSoftware Product Line Engineering (SPLE)Quality Assurance in SPLEStrategies and Techniques for Quality Assurance in Domain EngineeringConclusion and PerspectivesMotivationOne key aim of QA in SPLE–uncover defects / “bugs”A defect in a reusable artifactcan affect all / many PL applications–Example: ”Calling”is commonality in mobile phone PL -Defect in call component⇒Failures in all mobile phones–Example: Undesired feature interaction between variants (resp. features) “silent mode”and “soundalarm when battery low”⇒Feature interaction in all mobile phones with both variants“Excursion:”Feature Interactions in GeneralBob block callsfrom:AliceMotivationderived might have to becorrectedD o Q A i n S P LE a s e a r l y a sp o s s i b l e!D o Q A i n S P LE a s e a r l y a sp o s s i b l e!i.e.,a l r e a d y i n do m a i n e n g i n e e r i n g!i.e.,a l r e a d y i n do m a i n e n g i n e e r i n g!QA in Domain Engineeringexists in domain engineering that could be tested.E x is t in g Q A t e c h n i q u e s f r o m t h e d e v e l op m e n t o fs i n g le s o f t w a r e p r o d u c t s c a n n o t b e a p p l i e dd ire c t l yE x is t in g Q A t e c h n i q u e s f r o m t h e d e v e l op m e n t o fs i n g le s o f t w a r e p r o d u c t s c a n n o t b e a p p l i e dd ire c t l yStrategies for Handling Variability«component»:MP3Player «component»:PDFViewer«component»:UserInterfacePlayMediaOVMText V2Audio V3ImageV1AgendaSoftware Product Line Engineering (SPLE)Quality Assurance in SPLEStrategies and Techniques for Quality Assurance in Domain EngineeringConclusion and PerspectivesOverviewObjective:Create placeholders that abstract or simulate variantsin domain engineering is enabledeven if no variants have been realizedChallenges1.How to keep the effort for creating the placeholders to aminimum?–Creating placeholders requires development effort⇒Number of placeholders should be kept as small as possible2.How to guarantee an adequate coverage of the domainartifacts?–Variants are not checked when following the commonality strategy ⇒Complementary QA activities should be plannedExample Technique: “Testing”ScenTED[Pohl & Metzger, 2006; Reis et al. 2007; Reuys et al. 2005]–Model-based testing technique for system and integration testing of product lines–Developed in the European research projects ESAPS, CAFÉ, andFAMILIES [van der Linden 2002]–Experience of test engineers and test managers [Goetz et al. 2005; Reuys et al. 2005]:-ScenTED significantly supports the derivation of reusable systemtest cases in product line engineering-The systematic derivation of test cases from domain requirements leadsto product line requirements specifications of higher qualityExample Technique: “Testing”4: Derive Test CasesApplicationTest CasesExample Technique: “Testing”1122Test case scenarios are generated such thatCommon functionality of the software product line is coveredNumber of placeholders is minimized ( Challenge 1)ScenTED supports reuse of test cases for application testing This complements test of the commonalities by performing product-Challenge 2)OverviewObjective:(b) Apply QA techniques from the development of single systemsChallenges1.How to determine representative sample products?–Checking sample products should allow drawing conclusions about the overall quality of the PL2.How to keep the number of selected sample productsmanageable?–Number of sample products should be kept as small as possibleExample Technique: “FI Analysis”[Calder et al. 2003]Feature Interaction Analysis with RAFINA [Metzger et al. 2005]–Assumptions for selection of sample products:1. Interaction between features f 1, …, f r⇒Interactions between all features f 1, .., f q with 1 < q < r-i.e., there will be no m -way feature interactions (with m > 2)-m -way feature interaction := feature interaction that……does not occur between 1 < i < m features…but occurs among m features [Kawauchi & Ohta, 2003]2. Each variant is related to a feature ( X-Link)–Select sample products which bind the maximum number of variants⇒Feature interactions uncovered in a sample productwill be present in all “smaller”products (with a subset of the interacting variants bound) ( Challenge 1)Example Technique: “FI Analysis”Violation of Assumption 1:–Uncovered FI is in fact an m-way interaction (m> 2)⇒No feature interactions between sub-set of features⇒RAFINA will deliver a false positive result in this casem-way interactions are sparse–Some in the telecom domain (cf. [Kawauchi und Ohta2003])–None in our case studies in the embedded systems domain (cf.[Metzger, 2004])⇒Number of false positive results is small / negligibleExample Technique: “FI Analysis”…V nExample Technique: “FI Analysis”k1213 : Significantly higher reduction compared to brute forceExample Technique: “FI Analysis”feature interactions could be uncovered, Check if combination of feature interactions is allowedB u t: M o d if ic a ti o n o f a r ti fa c ts m ig h t n o t b e p o s s ib le !B u t: M o d if ic a ti o n o f a r ti fa c ts m ig h t n o t b e p o s s ib le !Example Technique: “FI Analysis”Solution 2: Only check pair-wise feature-combinations –Number of variants a⇒a(a −1) sample products–Drawback:-m-way interactions with m> 2 won’t be detectedExample: Comparison with “initial”RAFINA–100 VPs w/ 3 variants each300 ⋅299 = 89,700 sample products–“Initial”RAFINA (k= 2):-K(2, 3) = 3 3100≈1047sample productsrepOverviewObjective:Although sample strategy allows checking all variants byvariants are not checked in all potential reuse contexts,Challenge1.How to handle the complexity in checking all PLproducts?–Number of products of industry-relevant PLs prevents brute-force approach-Example:-15 variation points with 2 variants ⇒1 billion PL products –Industry reports on PLs with > 10.000 variation points and variantsExample Technique: “Formal Reasoning”Formal Verification with AVIP [Metzger et al. 2007; Metzger 2007;Lauenroth & Pohl, 2007]–Identify inconsistencies in reusable artifacts by formal reasoning–Key to handling the complexity:-State-of-the-art SAT solvers-Inspired by [Czarnecki & Pietroszek, 2006]and [Thaker et al. 2007]–Approach:1.Express artifacts and constraints as propositional formula2.Feed formula into SAT solvere SAT to compute whether the formula has a solutionExample Technique: “Formal Reasoning”1..*AudioV3┌O ┐=(V1 ∨V2 ∨V3)Example Technique: “Formal Reasoning” Derive propositional formula (cont’d):MediaPlayer«component»W1:MP3Player«component»W2:PDFViewer«component»:UserInterfacePlayMedia┌A┐=(W1 ∧¬W2) ∨(¬W1 ∧W2)Example Technique: “Formal Reasoning”are replaced by propositional no violation for valid PL productExample Technique: “Formal Reasoning”Component Diagram«component»MediaPlayer«component»W1:MP3Player «component»W2:PDFViewer«component»:UserInterfacePlayMediaUserConsoleOrthogonal Variability Model (OVM)VP1:MediaVP Text V2Image V11..*AudioV3┌A ┐=(W1 ∧¬W2) ∨(¬W1 ∧W2)┌O ┐=(V1 ∨V2 ∨V3)1..1V3) ( (V1 V2 ∨V3) V2)) )┌( (V3 (V1 ∨(V3 ∧∨V2)) )┌G ┐= ¬( (V1 ∨V2 ∨V3)⇒( (V3 ∧¬(V1 ∨V2)) ∨(¬V3 ∧(V1 ∨V2)))G ┐= ¬( (V1 ∨V3)⇒( (V3 ∧¬∨V2)) ¬(V1 V2)))AgendaSoftware Product Line Engineering (SPLE)Quality Assurance in SPLEStrategies and Techniques for Quality Assurance in Domain EngineeringConclusion and PerspectivesConclusionQA in SPLE is an important field of research Three strategies for QA in domain engineering:–Commonality Strategy–Sample Strategy–Comprehensive Strategyeach strategy poses specific challengeswhich must be addressed by QA techniquesFocus of Talk: Analytical QA techniques –Constructive QA techniques shouldbe used in addition-Examples:-code-generators or configurators formodel-driven product line developmentPerspectives“Debugging”for Comprehensive Strategy–Techniques give “counter-example”for consistency violation–Reasons for violation (actual defects) have to be located manuallyAutomated techniques for “debugging”are needed!Empirical Evaluation of Comprehensive Strategy –Techniques efficiently address the problem of complexityEffectiveness (ability to uncover defects) needs further investigation!Transfer strategies to service-based systems–Complexity problem in QA for service-based systems-Loose coupling of services allows for a potentially unbound numberof different systemsCan SPLE strategies & techniques for handling the complexity be applied to the complexity problem in QA for service-basedsystems?Further InformationContact me at:–andreas.metzger@sse.uni-due.deArticle–Pohl, Metzger:Software Product Line Testing:Exploring Principles andPotential Solutions.CACM, 49(12), 2006Web-Site–www.sse.uni-due.de/Text Book–Pohl, Böckle, van der Linden:Software Product Line Engineering ─Foundations, Principles and Techniques.Springer, 2005[Calder et al. 2003] Calder, M., Kolberg, M. Magill, E. et al., Feature interaction: a critical review and considered forecast, Computer Networks 41 (2003) 115-141.[Czarnecki & Pietroszek, 2006]Czarnecki, K., Pietroszek, K.: Verifying feature-based model templates against well-formedness OCL constraints. In Jarzabek, S., Schmidt, D.C., Veldhuizen, T.L., eds.: Generative Programming and Component Engineering, 5th International Conference (GPCE 2006), 22-26 October 2006, Portland, Oregon, USA, Proceedings, ACM (2006) 211-220 [Kawauchi & Ohta, 2003]Kawauchi, S., Ohta, T.: Mechanism for 3-way feature interactions occurrence and a detection system based on the mechanism. In Amyot, D., Logrippo, L., eds.: Feature Interactions in Telecommunications and Software Systems VII (FIW 2003), 11-13 June 2003, Ottawa, Canada, Proceedings., IOS Press (2003) 313-328[Lauenroth & Pohl 2007] Lauenroth, K.; Pohl, K.: “Towards Automated Consistency Checks of Product Line Requirements Specifications”, In: Proc. of the 22nd IEEE/ACM InternationalConference on Automated Software Engineering (ASE), Atlanta, GA,USA, November, 2007. [Metzger et al. 2005]Metzger, A., Büuhne, S., Lauenroth, K., Pohl, K.: Considering feature interactions in product lines: Towards the automatic derivation of dependencies between product variants. In Reiff-Marganiec, S., Ryan, M., eds.: Feature Interactions in Telecommunications and Software Systems VIII (ICFI'05), 28-30 June 2005, Leicester, UK, IOS Press (2005) 198-216 [Metzger et al. 2007]Metzger, A., Heymans, P., Pohl, K., Schobbens, P.Y., Saval, G.:Disambiguating the documentation of variability in software product lines: A separation of concerns, formalization and automated analysis. In Sutcliffe, A., ed.: 15th IEEE International Conference on Requirements Engineering (RE 2007), 15-19 October 2007, New Delhi, India, Proceedings, IEEE Computer Society (2007)[Metzger, 2007]Metzger, A. ”Quality Issues in Software Product Lines: Feature Interactions and Beyond (Invited Talk)”, In: F. Ouabdesselam, L. du Bousquet, Proceedings InternationalConference on Feature Interactions in Software and Communication Systems (ICFI 2007),Grenoble, France, 3-5 September 2007[Metzger, 2004]Metzger, A.: Feature interactions in embedded control systems. Computer Networks 45(5) (2004) 625-644[Pohl & Metzger, 2006]Pohl, K., Metzger, A.: Software product line testing. Communications of the ACM 49(12) (2006) 78-81[Pohl et al. 2005]Pohl, K., Böckle, G., van der Linden, F.: Software Product Line Engineering: Foundations, Principles and Techniques. Springer, Heidelberg (2005)[Thaker et al. 2007]Thaker, S., Batory, D., Kitchin, D., Cook, W.: Safe composition of product lines.In: Generative Programming and Component Engineering, 6th International Conference (GPCE 2007), 1-3 October 2007, Salzburg, Austria, Proceedings, ACM (2007)[Reis et al. 2007] Reis, S; Metzger, A.; Pohl, K.: “Integration Testing in Software Product Line Engineering: A Model-Based Technique”, In: Dwyer, M.B.; Lopes, A. (Eds.) ProceedingsFundamental Approaches to Software Engineering, 10th Intl. Conference, FASE 2007, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2007, Braga, Portugal, March 24 -April 1, 2007; LNCS 4422. LNCS, March 2007, pp. 321-335.[Reuys et al. 2005]A. Reuys, E. Kamsties, K. Pohl, S. Reis: Model-Based System Testing of Software Product Families. In: Advanced Information Systems Engineering, CAiSE2005 (Porto, Portugal, June 2005). LNCS, 3520, Springer, Heidelberg 2005, 519-534.[van der Linden 2002] F. van der Linden. Software Product Families in Europe: The Esaps& CaféProjects, IEEE Software, 19(4), 2002, 41–49.。