测试质量衡量标准（Test quality measurement standard）Quality scale (scale)Product quality can be clearly quantifiedTest coverage - code block coverage, function coverage, use case coverage... So much coverage, per coverage, what is the reasonable target? 50%, 80%, 100%According to the number of defects found, how much is found by the user, how much is found by the internal non test team, and how much is found by the test team, which is one of the measures of qualityRecurrent number of recurrent defectsPatches and Service package quantities are used to measure qualityWe have so many standards that can be used to measure quality. So what should be the core standards, the most important universal standards? How do you relate the standards to the quality?Setting out quality indicators and what is the correct indicator can guide us to decide whether to release or delay the release of the product until we reach the targetHow do you define testing efficiency, including how to measure the impact of s changes on testing?How do you define testing completion?Complex field product testing:Audio and video quality testing"Does it look effective?""Does that sound effective?"Is the effect good?Testing and judgment of various subjective typesThe impact of testing tools on the system itself (uncertainty principle):The uncertainty effect of the performance test tool itself on the machine performanceHow do you determine the end of a test point for a software?Before the software dies, if there is no end point of the test, then the software test will never end, and it will never end. The end point of software testing, according to the specific circumstances of their own company to develop, can not be generalized! Personally, the testing end point is determined by the following conditions:1. based on the "test phase" principle:Each software test usually goes through unit testing, integration testing and system testing. We can set up detailed test end points for unit test, integration test and system test respectively. Each test phase meets the end criteria and then tests at the latter stage. For example: unit testing, we require testing end points must satisfy the "100% core code by Code Review", "functional coverage rate reached 100%, the coverage rate of not less than 80% lines of code, there is no A and B defect, all defects found at least 60% are included in the defect tracking system and all levels of defect repair rate reached the standard of" standard. The end points of integration testing and system testing are formulated as related end standards, and so are of course.2., based on the principles of test cases:The test designer designed the test cases and asked the members of the project team to participate in the review. Once the test case was approved, the latter test could serve as a reference for the end of the test. For example, if the test case pass rate is too low during the test, you can refuse to continue testing until the developer fixes it. The pass rate of function test cases is 100%, and the non functional test cases reach more than 95%, which allows the normal end of the test. However, it is critical to grasp the quality of the test cases when using this principle as the end point of the test.3., based on the principle of "defect convergence trend":The life cycle of software testing with the test time, defect map line test found first, gradually increasing, and then testto a certain stage, and defects of a downward trend, until the discovery of almost zero defect or difficult to detect defects so far. We can determine whether the test can end through the tendency of the trend line of the defect, which is also a criterion.4. based on the principle of defect recovery rate:Software defects are divided into several levels in the test life cycle. They are: serious errors, major errors, minor errors, general errors, minor errors, and 6 suggestions for testing. Then we test in determining the end point, serious mistakes and main defect repair error rate must reach 100%, does not allow the existence of functional errors; minor errors and wrong defect repair rate must reach more than 85%, allowing the presence of a small amount of functional defects, solve the defect repair for later versions; low error best rate reached 60%~70% above. For the proposed test questions, you can not modify it for the time being.5. based on the principle of acceptance test:Many companies are doing project software, if this is to determine the end of the test point, it is best to test a certain stage, to reach or close to the test department designated standards, then submit the user acceptance test. If the user acceptance testing, you can immediately terminate the test department of the test; if the customer acceptance tests, and found some defects, it is possible to modify the defects of, after verification by the client, the corresponding test can be ended.6., based on the principle of coverage:For the test "coverage" principle, personal feel as long as the test case coverage "covers all the customer software requirements, including industry hidden demand, functional requirements and performance requirements and so on, as long as the execution of a test case coverage rate reached 100%, basically can end test. For example, the minimum coverage of a sentence in a unit test should not be less than 80%, the test case execution coverage rate should be 100%, and the test coverage requirement should be 100%, which can be used as the end of the determination point. If you're not sure, you have to look at the execution of the test case and check if the use case is missing. You can do sample testing and random testing of commonly used functions". For coverage, unit testing, integration testing, and system testing can not be ignored at each stage.7. based on the "project plan" principle:In most cases, each project will be prepared to develop and test from the beginning of the Schedule, in the corresponding test plan will also be a corresponding milepost, restrictions on the test schedule and test end point, will generally and project team members (development, management, testing, marketing, sales) to reach a consensus after the team agreed to develop a standard end point. If one aspect of the project is delayed, the test time is shortened accordingly. In most cases, all the required test content and regression tests have been completed and can serve as an end point. Many non-standard softwarecompanies are planning the project as a test end point, but if it is used as an end point, the test risk is greater, the quality of software is difficult to guarantee.8., based on the principle of defect measurement:This principle may not be used by many people, but relatively little. We can use the commonly used defect analysis techniques and defect analysis tools to count the defects that have been found. We can use the chart to count the data. It is convenient to consult and measure the defects in different periods. I remember that zhuzx put forward the problem of defect analysis technology in this forum, and I will not repeat it. We can also refer to the "test period, defect density" and "run time defect density" as an end point. Of course, the best criterion for testing ends should be that the number of defects is within an acceptable range". For example, ten thousand lines of code at most allow the existence of any number of serious errors, which are better quantified, better implemented, and become the mainstream of test defect measurement.9. based on the "quality cost" principle:A software often stops with balance between three aspects of quality / cost / schedule. As for these three areas, which one occupies the main position, it depends on what software it is. For example: human life is the aerospace software, then the quality is more important, even spend more money, also want to postpone the progress of testing can ensure high quality before the termination of the test, release. If it is common software, because of the interests and market reasons, even if there arebug, but also have to launch products, there is no way ah. In general, the primary reference is to "balance the costs of finding defects and the losses that this defect may cause."". Specific operations, you can according to the actual situation of the company to define what kind of circumstances is "the cost of testing the most cost-effective, most reasonable", while ensuring that the interests of the company to maximize. If the cost of finding a bug is higher and the user finds that the cost of bug is still high, the test can also be terminated.10. principles based on "testing industry experience":In many cases, some experience in the testing industry can also serve as a reference for our testing. For example, the testers' familiarity with the business, the ability of the tester, the efficiency of the test, and so on, will affect the implementation of the overall test plan. If a test team, everyone has no experience in industry project data accumulation, get a new project, nature is confused, don't know where to start, the test is not high quality natural. Therefore, through the tester's experience, the verification test execution and the end point will also play a key role.Various combinations of test elements (large testing range):Test element combinations, covering various possible combinations, will become large: operating systems, vs. debugging / publishing, vs. hardware configurations, vs. languages, vs., etc., vs., etc.Unlimited numbers of users may enterTests for products that have time correlation. The possible exhaustion of various times is infiniteProblems throughout the product range testStress test for the whole productThis product performance tests vs. each development team's performance tests on its own modulesIntegration testingTest set optimization:Determined by time and schedule impact?Determined by the user influence?Determined by the number of defects found in the average test case (or by considering other ROI)Select test cases that cover the changed code, and so decide?Determined by the complexity of the code to be testedProject plan arrangement:Accurately estimate the time required for testingHow does the test team participate in determining the overallschedule of the project?Planning for agile, rapid, and iterative testingTest impact on the project:For repairing defects such as i.e. development group the defect and they answered "no one will do so!", this time how well insist on repairing defects.Test team participation in design phase - testability analysis / designShould I have an impact on the decision to publish / not publish?Test automation:The late maintenance nightmare of automated test casesHow do you simulate human eyes and ears to do automated testing (audio / video testing)?There is not enough interface in the product code to support automated testing (such as controls drawn by developers themselves)Automated testing of simulated N user actions (N is very large)Simulate real users - [random user behavior]Integration test:Automated testing in integrated testingDebugging responsibilities, who do integration testing, who is responsible for debugging the entire product issue?What are the test cases that integration tests should include?Other general problems:After several releases, the accumulated test code became bloated and difficult to maintainPoorly designed test code, repeated test code, lack of overall design and architecture among the various test automation teams, and avoid redundant workRedundant test casesRetaining experienced test talentMeasurement of software testing process1) the role of test metrics (-)A: provides the basis for developing test plansHow long will it take? What material conditions do you need? How many people and what qualities do you need? How much can be accomplished within a given time?Which modules and functions need to be focused on? Test effort accounted for the proportion of the project What goals can we achieve after the test? Wait(these data are some of the necessary reference values for the development of test plans during project initiation, especially in the planning of resources. Different projects may have its particularity, but in general, they still have some rules can be found, the past empirical data can be used as an estimate, if the project experience, it can be found from the historical data and new projects of a similar situation, to be more accurate to complete the program. )B: improve test process controllabilityImprove test efficiency and qualityImprove testers' sense of achievement2) which process is testing to measure?(product early market evaluation, test strategy analysis, testability requirement analysis, test tool analysis, use case design phase, execution phase and FOA stage)We need to do metrics at several critical stages of the test, which are: use case design phase, execution phase, and FOA phase. Test case design stage includes test scheme is determined, test tool design, test case design, test execution stage obviously, the process of our testing, such as integration testing, system testing, performance testing, regression testing, measurementof work also includes developers complete unit test. The FOA phase is the first step in testing quality. Through the FOA, we can obtain many metrics that contribute to the quality of the product. This is also a measure of the value of the test. Almost all of the testing process seems to have been included. In fact, this includes only the specific stages of testing.3) the content of the test metricsTwo measurement types:A: project metrics: scale, test effort, test progress, test productivityB: quality measurement: defect rate (stage), defect removal rate, reliability, etc.Four basic metrics: size, workload, schedule, and defects4) metrics in the design phase of the test caseA: size: test plan quantity, test case number, test tool design quantity, test case / person monthB: workload: the draft of the document, writing the workload, review the reading workload, review the workload, modify the workloadC: schedule: the beginning, end time, actual start end time, planned working hours, actual working hours and planned completion rate of each specific workD: defects: errors, quantities, defects, quantities, and grades occurring during the review process5) test execution phase metrics:Test case execution rate and test case passing rateTest case, problem discovery rate, BUG numberBUG level statistics? BUG distribution statistics (modules)BUG distribution statistics (stages) BUG densityThe BUG closure rate, per capita BUG, was found to be efficientTest case execution workload, project regression test execution effortNumber of published documents, published documents, defects, quantities, and gradesField number of BUG was found and regression tests were performed on field BUG workloadThe validation cycle during the release process; the validation effort during the releaseTest case coverage, function user concernsDemand change level6) test metrics contribute to project implementationMeasure, meaning, purpose and meaningTest productivity, unit time, amount of code to be tested, or number of test cases per unit time, a team's testing capabilityThe percentage of actual work spent relative to the estimated amount of work relative to the estimated amount of work, improved estimation skills, and avoidance of overload assignmentMonitor the progress rate of the project, the deviation of the project's actual test progress relative to the planned progress, and monitor the project so that corrective actions can be taken in due courseThe amount of work done by the workload test; the amount of work that the test contributes to the whole projectThe defect density, the number of defects found in thousands of lines, and the number of defects found by thousands of functions, are used to measure the reliability of the system under testThe test of the severity of the problem revealed that the severity of the problem was distributed to determine the reliability of the currently tested systemThe problem of test cases, the discovery of efficiency, and thenumber of problems found in a single test case are used to measure the validity of the test caseTest case coverage, demand coverage, function point coverage, code coverage, etc., measure the adequacy of the testThe issue of missing rate, the amount of market feedback after issue, the total number of product / product problems, and the quality of internal testingCOQ the amount of work needed to improve the quality of the testCOPQ pays for bad quality7) who will do the measurement?8) how do you measure?PDCA method:The first step: Plan (plan, set the benchmark (plan), making our desired goals)The second step: do (Implementation) (Daily - - recording data) (weekly - - summary data, giving the results of the measurement)The third step: check (check and post what (GAP) - weekly meetings for the measurement results, make the next work suggestion)The fourth step: action (improvement process) (stage summary- subsystem, integration, system testing and other test phase after the end of the measurement, evaluation, for follow-up work)Make directionThe fifth step: return, to, plan\AutoRunner software testing toolsTestCenter powerful test management toolQACenter-- software black box testing tool。
