a3_leisgang_paperSNUG Germany 2010 User_Papers
- 格式：pdf
- 大小：419.74 KB
- 文档页数：17
“What gets measured gets done” -Predictable verification with VMM PlannerTobias Leisgangt-leisgang@Texas InstrumentsFreising, GermanyABSTRACTWith growing complexity of designs and verification environments, verification plan-ning and tracking became cumbersome. Verification plans were overloaded and links between specification, plan and the verification environment have been missing. Fur-thermore progress tracking was based on ballpark figures. This paper shows how to at-tack these limitations with the help of VMM planner, ensuring your verification projects completeness and make it more predictable.Table of Contents1 INTRODUCTION (3)2 CREATING A HIERARCHICAL VERIFICATION PLAN (3)2.1 M OTIVATION FOR A HIERARCHICAL VERIFICATION PLAN (3)2.2 W HY VMM PLANNER FITS OUR NEEDS (4)2.2.1Attributes (5)2.2.2Metrics (6)2.2.3Features (6)2.3 V ERIFICATION PLAN BASED ON A SPREADSHEET (6)2.3.1Plan sheet (6)2.3.2Attributes sheet (7)2.3.3Metrics sheet (7)2.4 V ERIFICATION PLAN WITH DVE COVERAGE (HVP PLAN) (7)3 ADDING METRICS AND BACK-ANNOTATING DATA (10)3.1 A DDING METRICS TO A SPREADSHEET (10)3.2 A DDING METRICS IN DVE (10)3.3 B ACK-ANNOTATE REGRESSION DATA (12)3.4 C REATE VIEWS (14)3.5 M ILESTONE BASED VERIFICATION TRACKING (15)4 IMPROVEMENTS (16)5 CONCLUSIONS (16)6 REFERENCES (17)Table of FiguresFigure 1 VMM planner flow (5)Figure 2 Plan sheet (7)Figure 3 Attribute Sheet (7)Figure 4 Simple HVP plan in DVE (8)Figure 5 Built-in metrics (9)Figure 6 Adding attributes to a feature (10)Figure 7 Linking verification database to the plan by drag-and-drop (11)Figure 8 Completeness checks (12)Figure 9 Backannotated spreadsheet (13)Figure 10 Coverage report with HVP back-annotated (14)Figure 11 Detailed report for a feature (14)Figure 12 Release REL08 view (16)Table of TablesTable 1 Examples of user-defined attributes (6)1 IntroductionConstraint-random verification methods became a must when complexity of designs steadily increased over the last years. Needless to mention that verification environment complexity grew accordingly. As a consequence the number of coverage items, coverage crosses, checks and testcases increased. This led to huge text documents, spreadsheets or something comparable that traditionally is used for verification planning. To make it even more complicated several documents exist to plan the verifica-tion environment.Over time this methodology revealed various weaknesses. Translating specifications into lists of cov-erage items and crosses led to overloaded spreadsheets and at the end of the project the link to the spe-cification was lost. The bigger the plans became, the harder it was to judge if a particular feature is represented with coverage and appropriate checks. In addition the risk to miss items from the plan in the verification environment creation increases with the number of items in the plan.Another aspect was tracking of the verification progress. While this was easy in times of directed tests, where a testcase corresponded with a particular feature, this is not longer trivial for a constraint-random methodology. What's the degree of testbench completion? Is verification almost completed when your coverage is close to 100% and everything is coded? Does it mean only corner case bugs are left? More than once the most critical bugs have been found in this phase and one week to completion became two months to completion.This paper will depict how we attacked these weaknesses with the help of VMM planner. In the first part we'll show how a hierarchical verification plan can look like and how to create it, either as a spreadsheet or with DVE. After that you will see how metrics can be added to particular features, how to link your regression data to features and how to create reports from that data. After touching custo-mization of these reports, the paper will give an outlook what can be improved on the methodology and the tools used.2 Creating a hierarchical verification plan2.1 Motivation for a hierarchical verification planFirst step in a verification project is creating a verification plan. In all of our previous projects this was mainly a coverage plan. The verification engineer studied the specification and came up with a more or less long list of coverage items. After this a big bunch of crosses has been added to make sure all poss-ible combinations are exercised and verified. The motto was: The more crosses, the better. Doing this we had maximum confidence. When module complexity began to grow we noticed that also a checker document is indispensable. After both documents have been completed they got reviewed and the cod-ing phase was started. The coverage items have been translated into verification environment code and verification progress was assessed by looking at the total coverage number. Who hasn’t been in the situation that total coverage was 99% some days before the tapeout. The remaining holes are inside some really huge crosses. So we are quite sure this is only rare corner cases and feel very confident fortapeout. While silicon is produced the remaining holes are filled and it turns out, that one of the new features intended for the lead application is not working under a certain condition. Could it go worse? So after some of these epiphanies, the limitations of this approach became clear. We do neither sell coverage nor checkers to a customer, we sell fea tures. So wouldn’t it be natural to create a verification plan that is based on features? But as stated in the beginning, the number of features grew dramatical-ly. So there’s a need to create a plan in a hiera rchical manner. Otherwise we would get lost.Another hurdle is that our specifications aren’t the final ones when the verification plan gets reviewed. Maybe it is unique to our company, but maybe some of you face the same misery. So incremental up-dates to the specifications can easily be missed in the coverage plan. And even if it was put to the cov-erage plan, it can be left out while coding. We would not notice it as we only look at the coverage re-port to assess our plan completion. Having all planned testbench items coded and running regressions, we check to which extend the design is exercised and if all required stimulus is applied. But the as-sessment of verification progress based on total coverage might be misleading. The most important feature might be within the last 1% coverage.Having these limitations in mind we had a list of requirements on future verification planning metho-dology. First of all, planning should be based on features from the specification. The plan should sort them in a hierarchical manner to prevent we get lost in a huge feature list. But of course there must be a link to coverage, checks and tests. To track the verification progress it is essential to have the cover-age report mapped to the plan to assess the amount of features that are exercised. This also serves as double check if everything from the plan is coded.2.2 Why VMM planner fits our needsHaving all these requirements in mind we tried to address them with our first VMM project that we started. This was about two years ago where we verified an USB subsystem in a microcontroller. While trying to get familiar with the concepts of VMM we also came across the VMM planner tool. This new tool was still in beta stage, but promised to fulfill all of our requirements. The VMM planner flow is shown in Figure 1.Figure 1 VMM planner flowThe basis of VMM planner is a HVP plan. This is a verification plan written in a XML like language called HVP. You don’t have to learn this language. T here are several user interfaces available. Either you create your plan with a XML spreadsheet, Word document or DVE planner. Two of them are de-scribed in chapters 2.3 and 2.4. The VMM Planner application takes the HVP plan and a unified cov-erage database as inputs and links them together. By that you can track your verification plan comple-tion. In addition there’s the opportunity to process data from other sources and link them to a plan. You can think of a database that contains pass/fail information for testcases in a regression.A typical HVP plan consists of attributes, metrics and mainly features. Let’s have a look at them in detail.2.2.1 AttributesAttributes are used to describe features. You can describe as many attributes as you want. This de-pends on your needs for the verification plan. Attributes can be either a number (integer or real) or a text value (string or enum). VMM planner already has some built-in attributes like defining an owner.I f you’re fine with them there’s no need to define attributes on your own. Table 1 shows a list of user-defined attributes we used in our plan.And here’s how this translates in the HVP plan:attribute string coveragenotes = "";attribute enum{REL01, REL05, REL08, REL10} release = REL10;attribute integer spec_page = 1;2.2.2 MetricsMetrics define values from a verification database that should get annotated to features. They do not create the link between a feature and a coverage item. This is done by measure described in chapter 2.2.3. VMM planner also offers a set of built-in metrics like line coverage, group coverage or test re-sult.2.2.3 FeaturesThis is the main part of the plan. Features are arranged in a hierarchical manner in a feature tree. There’s no limit in size and depth of the feature tree. Each feature has associated attributes and meas-ures. Measures indicate the corresponding metrics in the verification database. If you measure the completion of feature A by functional coverage and the respective coverage group is named A_COV, the measure would be of type group and the source would be the path to coverage group A_COV in the verification database.2.3 Verification plan based on a spreadsheetFor the first verification plan we created, the only available user interface was a XML spreadsheet. Since we didn’t intend to write our plan in HVP language we chose this opt ion. Special attention needs to be taken to the tool for creating spreadsheets. Since Microsoft Excel and the OpenOffice spread-sheet application handle XML spreadsheets in different manner, it was not possible to exchange them easily between the tools. This might have changed in the meantime, but you should try it first before you build a plan with thousands of features. A VMM planner spreadsheet file consists of three sheets that correspond to the three basic elements of a HVP plan:2.3.1 Plan sheetThis sheet contains the feature description. An example can be seen in Figure 2. The first cell defines the name of the plan. Column two and three are the feature and subfeature definitions. The number of columns defines the hierarchical depth of the plan. The next three columns are used to describe thefeature in more detail and define during planning which coverage items and checks are required to verify this feature. The owner column is used to indicate who will take care of the particular feature. The column release defines at which milestone the feature needs to be verified. This will be explained in more detail in chapter 3.5. The measure column defines the link between the verification database and the plan. In the last column we will later see the back-annotated value.2.3.2 Attributes sheetThis sheet contains user-defined attributes which we described in chapter 2.2.1. In Figure 3 every row describes an attribute. In the name column we note the name of the attribute. The third column defines if the attribute is of type integer, real, enum or string. In the fourth column we can set a default value for an attribute. The column propagate defines if an attribute propagates to the children of the features it is assigned to.2.3.3 Metrics sheetThis sheet contains the user defined metrics. Since we were happy with the built-in metrics we left this sheet empty.2.4 Verification plan with DVE coverage (HVP plan)When we started the next project the possibilities to create a verification plan have been extended to the DVE GUI. Since the spreadsheet didn’t support the co ncept of arranging features in a hierarchical manner very well, we decided to try this option. Although there are some limitations like missing sup-port for enum attributes or no search/replace capabilities, creation of a plan was very easy. In the file menu you create a new HVP plan. Figure 4 shows how a simple HVP plan can look like. In the left pane you see the plan. In the right pane you can define user-defined metrics. Make sure you have the plan selected in the left pane, otherwise you define the attributes only for a particular feature. To addmetrics you do the same on the metrics tab. Also here we only use the built-in metrics which you can see in Figure 5.Figure 4 Simple HVP plan in DVEOnce you’ve defined your attributes and additional metrics, you can go and build up your feature tree. Each feature inherits the global attributes with their default value. But you can easily overwrite them in the right lower pane with the value you need for this particular feature. An example is shown in Figure 6.Figure 5 Built-in metricsFigure 6 Adding attributes to a feature3 Adding metrics and back-annotating dataOnce your verification plan is ready you will start verification environment creation, code your cover-age items and checks and add testcases to your regression. Now it is time to link your verification plan with the coded coverage, checks and tests to track your verification progress. The next two paragraphs will show how to do that for the two plan types described in chapter 2.3.1 Adding metrics to a spreadsheetAdding metrics to a spreadsheet is a manual process. In the columns where you noted the measures you add the reference for the respective item in the verification database. An example for a coverage item is shown in Figure 2 in the column measure.3.2 Adding metrics in DVEAdding metrics in DVE is even simpler. One way would be to add them manually like in the spread-sheet. The other way is linking them based on a verification database that is loaded into DVE. Regard-less of the option you use, you have to add measures to features first. This is very simple, you right-click on a feature and add a measure. Now you can define the metrics of the feature manually and addthe link to an element of a verification database. If you do it manually you would add the path to the respective item like in the spreadsheet.But the second alternative is way smarter. In addition to the loaded HVP plan, you load a verification database from a single test or your first regression. This is done by loading a database in the file menu. Once this is done you should see two panes like in Figure 7. In the upper pane you have your verifica-tion plan with measures added to the features. In this example we have added the measure coverage to the feature Parity_bit, which definition you’ve seen in Figure 6. In the lower pane we see the loaded coverage database. There you can see the coverage class payload_cov with its single bins and cross coverage items. Now you can drag a coverage item from the lower pane to the Sources section of the measure on the right side. By that you already linked your coverage item to the respective measure of the feature. That’s very easy.Figure 7 Linking verification database to the plan by drag-and-dropSi nce there’s a huge chance to forget something in a plan with thousands of features, DVE offers two kinds of sanity checks. The first one indicates if a measure is defined, but has no metrics and/or source defined. This is indicated by a red exclamation mark next to the measure as you can see in Figure 8. The other completeness check indicates if there are items in the verification database that are notlinked to a feature. This can be on intention because you link a cover cross to a feature, but the cover points that build the cover cross are not linked. It could also indicate an error, because an item was coded based on the plan but we forgot to assign a measure to this feature. This completeness check is indicated by an orange star next to the item in the verification database.Figure 8 Completeness checks3.3 Back-annotate regression dataDuring the process of adding measures we already found out if everything from the plan is reflected in the verification environmen t. But we’re still missing one of our most important requirements. We want to know our verification progress. Now it’s time to run a regression and back-annotate the acquired data to our plan.Generating an annotated spreadsheet is very easy. All you need to do is to pass the location of your spreadsheet and your verification database to the HVP tool:hvp annotate --plan_in my_plan.xml --covdb my_database.vdb –lcaThis will output a back-annotated spreadsheet like shown in Figure 9. You can see that the last column is filled with the values from your verification database. If a feature tree consists of several subfeatures the parent feature is automatically annotated with the sum of all subfeatures.If you have generated your plan with DVE planner, you can even combine the merging of verification databases with the back-annotation. Both actions are performed by urg, so they can be combined in one step. To speed this up for a huge number of verification databases you can utilize the parallel op-tion and split the processing load on a LSF grid:urg -f dirs.f -parallel -report report_directory –dbname my_vdb -plan my_plan.hvp The dirs.f file contains a list of testcase directories where the verification databases are found. When the script has ended you get a coverage report in HTML format. The interesting thing is the HVP tag that shows you the back-annotated verification plan on HTML format. In Figure 10 you can see the plan named SIO with the feature tree. Every hierarchy is intended and the score for each metrics is shown. If there’s no metrics linked there will be a grey box.Figure 10 Coverage report with HVP back-annotatedNow you can browse through the plan and look at each feature from the plan in detail. In Figure 11you can see that we fully verified the Parity_bit feature that we have linked in chapter 3.2.Figure 11 Detailed report for a feature3.4 Create viewsAs we already mentioned, with increasing number of features also the verification plan and the numberof metrics grow. So the plan can get pretty huge and as such also the back-annotated plan which meansthere are a lot of features you need to browse through. Also a team of verification engineers might work on the project to get it completed in reasonable time. The single verification engineer wants to find out which features he needs to pay attention. For that you can create different views from your plan utilizing filters or overrides.There are a lot of use cases how to use the filter and override mechanisms. An example is to search for features for a specific owner:filter my_items;keep feature where owner == "Tobias";endfilterYou apply this filter to the plan by passing a modifier file (mod_file) to urg:urg -f dirs.f –mod mod_file -parallel -report report_directory –dbname my_vdb -plan my_plan.hvpWith this method every verification team member can create his own personalized verification plan.3.5 Milestone based verification trackingAs already stated in the introduction, one of our existing weaknesses was that a very important feature is verified very close to the end. Also progress tracking based on a total coverage number was more than cumbersome. For our design process we already had a release based design flow in place, so we decided to apply this to verification as well. There are different release milestones defined which de-fine the maturity of a design. Applying this principle to verification was a perfect fit. The basic fea-tures of a design should be verified first. If they are completed, verification reached the REL05 miles-tone. If all features are verified in almost all possible combinations this marks the REL08 milestones. And finally if everything is verified and we’ve exercised the design with error injection and stress tests it reaches the REL10 milestone. When creating the plan we already decide which feature needs to be verified before a specific milestone and we mark the feature with the respective release tag.This enabled us to apply the filter concept to the back-annotation of the plan and to check if we reached a certain release milestone. A filter rule looks pretty easy:filter REL08;keep feature where release == "REL0.5";keep feature where release == "REL0.8";remove feature where release == "REL1.0";endfilterFigure 12 shows an example of a back-annotated plan for a REL08 milestone. As you can see the total coverage has only a score of 99.83%, so there are still some items to fill. But the score for the verifica-tion plan already reached 100%, because we’ve filtered it for the release REL08 milestone. All fea-tures marked for this milestone have been hit, so we reached the total score of 100%.Figure 12 Release REL08 viewDepending on the complexity and duration of the project you can add more granularities to get an indi-ca tion if you’re still on track and hit the expected tim eline. This methodology can also fit for two pass designs where the first silicon is used to deliver early samples to a customer. In this case, the design is not expected to work under all corner cases, but it is about time to market. During creation of the plan you simply mark the features you want to be functional for the customer sample. Once you’ve verified all of them you’re ready to tape out the d esign.4 ImprovementsThere are only few tools in th e world which can’t be improved. This applies to VMM planner as well. The biggest limitations we noticed are the plan editing ca pabilities of DVE planner. There’s definitely a search and replace functionality missing. We always needed to create reports and apply filter rules to search for particular features. This is not very easy to use.For the report generation a more advanced HTML report would be very much appreciated. The VCS help pages in HTML are a great example of how this could look like. You can browse through the fea-ture tree in the left pane and look at the feature pages in the right pane. Also a search function in the HTML report would be very helpful.The third feature that would make VMM planner more valuable could be adding hyperlinks to specifi-cation documents. Of course you can add page references to your plan manually, but with the next re-vision of the specification your plan would need major rework, because one page was added. We could think of adding hyperlinks to anchors in a HTML specification or to a PDF document.So if you’re part of the VMM planner R&D team and read these lines, we would be happy if you will fulfill some of these wishes to make planning more comfortable and grow the fan base for VMM plan-ner.5 ConclusionsFinally we could prove that VMM planner can be used to plan your verification project on a feature basis and in a hierarchical manner. And the good news is that all information can stay within one plan without getting overloaded. You’re free to choose b etween doing it based on a document, spreadsheet or use the DVE GUI to create this plan. But you should be aware of the limitations as soon as you de-cide to go with one of them.Even more power adds the capability to back-annotate your plan with your regression results. This makes the verification progress more measurable and predictable. Back-annotation reduces the risk of missing items from the plan during environment creation as it closes the loop to the plan. Verification progress can be judged more easily because you don’t have to review different reports and sum num-bers manually. Creating different views makes it even more attractive. You can easily review your verification progress on intermediate milestones for completion of goals set in the beginning of the project. Although there’s still room for improvement on the capabilities of VMM planner, we succes s-fully used it on three verification projects.6 References[1]VMM Planner User Guide, Version D-2009.12[2]Nancy Pratt, “Are we there yet”, SNUG San Jose 2008[3]URG, Unified Coverage Reporting User Guide, Version D-2009.12。
