Framework for Accelerated Stress Benchmarking

  • 格式:pdf
  • 大小:20.93 KB
  • 文档页数:20

Network Working GroupINTERNET-DRAFTExpires in: April 2004Scott PoretskyQuarry TechnologiesShankar RaoQwest Communicati onsRay PiattCable and Wireles sOctober 2003Terminology for Accelerated Stress Benchmarking<draft-ietf-bmwg-acc-bench-term-01.txt>Status of this MemoThis document is an Internet-Draft and is in full conformance withall provisions of Section 10 of RFC2026.Internet-Drafts are working documents of the Internet EngineeringTask Force (IETF), its areas, and its working groups. Note thatother groups may also distribute working documents as Internet-Drafts.Internet-Drafts are draft documents valid for a maximum of sixmonths and may be updated, replaced, or obsoleted by otherdocuments at any time. It is inappropriate to use Internet-Draftsas reference material or to cite them other than as "work inprogress."The list of current Internet-Drafts can be accessed at/ietf/1id-abstracts.txtThe list of Internet-Draft Shadow Directories can be accessed at/shadow.html.ABSTRACTThis terminology document provides the terms to be used forbenchmarking a router or switch under Accelerated Stress conditions.Terminology is provided for performing this benchmark usingthe Contol Plane, Data Plane, Management Plane, and SecurityPlane of the Device Under Test. Terms are defined forConfiguration, Startup Conditions, Instability Conditions, FailureConditions, and Benchmarks.Poretsky, Rao, Piatt [ Page 1]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003Table of Contents1. Introduction (3)2. Existing definitions (3)3. Term definitions (3)3.1 Benchmark Planes (3)3.1.1 Control Plane (4)3.1.2 Data Plane (5)3.1.3 Management Plane (5)3.1.4 Security Plane (6)3.2 Configuration Sets (6)3.2.1 Control Plane Configuration Set (7)3.2.2 Data Plane Configuration Set (7)3.2.2.1 Traffic Profile (8)3.2.3 Management Plane Configuration Set (8)3.2.4 Security Plane Configuration Set (9)3.3 Startup Conditions (10)3.3.1 Control Plane Startup Conditions (10)3.3.2 Security Plane Startup Conditions (11)3.4 Instability Conditions (11)3.4.1 Control Plane Instability Conditions (12)3.4.2 Data Plane Instability Conditions (12)3.4.3 Management Plane Instability Conditions (13)3.4.4 Security Plane Instability Conditions (13)3.5 Failure Conditions (14)3.5.1 Unexpected Packet Loss (14)3.5.2 Unexpected Session Loss (14)3.5.3 Misrouted Packets (15)3.5.4 Access Denial (15)3.5.5 Errored Management Value (16)3.5.6 Security Plane Failure (16)3.6 Benchmarks (17)3.6.1 Success Threshold (17)3.6.2 Accelerated-Life Test Duration (17)4. Security Considerations (18)5. References (18)6. Author’s Address (18)7. Full Copyright Statement (19)Appendix 1 - White Box Benchmarks (20)Poretsky, Rao, Piatt [ Page 2]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20031. IntroductionRouters in an operational network are simultaneously configured withmultiple protocols and security policies while forwarding traffic andbeing managed. To accurately benchmark a router for deployment it isnecessary to test that router in operational conditions bysimultaneously configuring and scaling network protocols and securitypolicies, forwarding traffic, and managing the device. It is helpfulto accelerate these network operational conditions so that therouter under test can be benchmarked with faster test duration.Testing a router in accelerated network conditions is known asAccelerated Stress Testing.This document defines the Benchmark Planes used for the AcceleratedLife Benchmarking. These are the Control Plane, Data Plane,Management Plane, and Security Plane. For each plane, theConfiguration Set, Startup Conditions, and Instability Conditions.Terms are also defined for observing Failure Conditions and reportingBenchmarks. White Box benchmarks are provided in Appendix 1 foradditional DUT behavior measurements.2. Existing definitionsRFC 1242 "Benchmarking Terminology for Network InterconnectDevices" and RFC 2285 "Benchmarking Terminology for LAN SwitchingDevices" should be consulted before attempting to make use of thisdocument.For the sake of clarity and continuity this RFC adopts the templatefor definitions set out in Section 2 of RFC 1242. Definitions areindexed and grouped together in sections for ease of reference.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" inthis document are to be interpreted as described in RFC 2119.3. Term definitions3.1 Benchmark PlanesDefinition:The features, conditions, and behavior for the Accelerated StressBenchmarking.Discussion:There are four Benchmark Planes: Control Plane, Data Plane,Management Plane, and Security Plane as shown in Figure 1. TheBenchmark Planes define the Configuration, Startup Conditions,Instability Conditions, and Failure Conditions used for the test. Poretsky, Rao, Piatt [ Page 3]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003___________ ___________| Control | | Management|| Plane |___ ___| Plane || | | | | |----------- | | -----------\/ \/ ______________________ | Security || |<-----------| Plane || DUT | | ||--->| |<---| -----------| ----------- || || ___________ || | Data | ||--->| Plane |<---|| |-----------Figure 1. Router Accelerated Stress Benchmarking PlanesMeasurement units: N/AIssues: NoneSee Also:Control PlaneData PlaneManagement PlaneSecurity Plane3.1.1 Control PlaneDefinition:The Description of the control protocols enabled forthe Accelerated Stress Benchmarking.Discussion:The Control Plane defines the Configuration, StartupConditions, Instability Conditions, and FailureConditions of the control protocols used for the test.Control Plane protocols may include routing protocols,multicast protocols, and MPLS protocols. These can beenabled or disabled for a benchmark test.Measurement units: N/AIssues: NoneSee Also:Benchmark PlanesControl Plane Configuration SetControl Plane Startup ConditionsControl Plane Instability ConditionsControl Plane Failure ConditionsPoretsky, Rao, Piatt [ Page 4]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.1.2 Data PlaneDefinition:The data traffic profile used for the Accelerated StressBenchmarking.Discussion:The Data Plane defines the Configuration, StartupConditions, Instability Conditions, and FailureConditions of the data traffic used for the test.The Data Plane includes the traffic and interfaceprofile.Measurement Units:N/ASee Also:Benchmark PlanesData Plane Configuration SetData Plane Startup ConditionsData Plane Instability ConditionsData Plane Failure Conditions3.1.3 Management PlaneDefinition:The Management features and tools used for theAccelerated Stress Benchmarking.Discussion:A key component of the Accelerated Stress Benchmarking is theManagement Plane to assess manageability of the routerunder stress. The Management Plane defines the Configuration,Startup Conditions, Instability Conditions, and FailureConditions of the management protocols and features used forthe test. The Management Plane includes SNMP, Logging/Debug,and Statistics Collection.Measurement units:N/AIssues:NoneSee Also:Benchmark PlanesManagement Plane Configuration SetManagement Plane Startup ConditionsManagement Plane Instability ConditionsManagement Plane Failure ConditionsPoretsky, Rao, Piatt [ Page 5]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.1.4 Security PlaneDefinition:The Security features used during the Accelerated StressBenchmarking.Discussion:The Control Plane defines the Configuration, StartupConditions, Instability Conditions, and FailureConditions of the security features and protocols used forthe test. The Security Plane includes the ACLs, Firewall,Secure Protocols, and User Login.Measurement units:N/AIssues:NoneSee Also:Benchmark PlanesSecurity Plane Configuration SetSecurity Plane Startup ConditionsSecurity Plane Instability ConditionsSecurity Plane Failure Conditions3.2 Configuration SetsDefinition:The features and scaling limits used during the Accelerated StressBenchmarking.Discussion:There are four Configuration Sets: Control Plane Configuration Set,Data Plane Configuration Set, Management Plane Configuration Set,and Security Plane Configuration Set.Measurement units: N/AIssues: NoneSee Also:Control Plane Configuration SetData Plane Configuration SetManagement Plane Configuration SetSecurity Plane Configuration SetPoretsky, Rao, Piatt [ Page 6]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.2.1 Control Plane Configuration SetDefinition:The routing protocols and scaling values used for the AcceleratedLife Benchmarking.Discussion:Control Plane Configuration Set is shown in Figure 2 and specifiesthe Routing Protocols, Multicast, and MPLS configuration. Specificprotocols can be enabled or disabled for a benchmark test.Measurement units: NoneIssues: None____________ ____________ ____________| Routing | | Multicast | | MPLS || Protocols |___ | Protocols | __| Protocols || | | | | | | | ------------ | ------------ | ------------| | || | || \/ || ___________ || | | ||------->| DUT |<------|| |-----------Figure 2. Control Plane Configuration ModuleSee Also:Data Plane Configuration SetManagement Configuration SetSecurity Configuration Set3.2.2 Data Plane Configuration SetDefinition:The data traffic profile enabled for the Accelerated StressBenchmarking.Discussion:Data Plane Configuration Set includes the Traffic Profile andinterfaces used for the Accelerated Stress Benchmarking.Measurement Units:N/AIssues:NoneSee Also:Traffic ProfilePoretsky, Rao, Piatt [ Page 7]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.2.2.1 Traffic ProfileDefinitionThe characteristics of the Offered Load to the DUT used forthe Accelerated Stress Benchmarking.DiscussionThe Traffic Profile specifies the number of packet size(s), packetrate per interface, number of flows, and encapsulation used for the o ffered load to the DUT.Measurement Units:Traffic Profile is reported as follows:Parameter Units--------- ------Packet Size(s) bytesPacket Rate(interface) array of packets per secondNumber of Flows numberEncapsulation(flow) array of encapsulation typeIssues:NoneSee Also:Data Plane Configuration Set3.2.3 Management Plane Configuration SetDefinition:The router management features enabled for theAccelerated Stress Test.Discussion:A key component of the Accelerated Stress Test is the Management C onfiguration Set to assess manageability of the router understress. The Management Configuration Set defines the management c onfiguration of the DUT. Features that are part of theManagement Configuration Set include SNMP, Logging/Debug, andStatistics Collection, and services such as FTP, as shown inFigure 3.Measurement units:N/AIssues:NoneSee Also:Control Plane Configuration SetData Plane Configuration SetSecurity Plane Configuration SetPoretsky, Rao, Piatt [ Page 8]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003____________ ____________| | | Logging/ || SNMP | __| Debug || | | | |------------ | ------------| || |\/ |___________ || | || DUT |<---|| |-----------||\/___________| Packet || Statistics|| Collector || |-----------Figure 3. Management Plane Configuration Set3.2.4 Security Plane Configuration SetDefinition:Security features and scaling enabled for the Accelerated StressTest.Discussion:The Security Plane Configuration Set includes the configurationand scaling of ACLs, Firewall, IPsec, and User Access, as shownin Figure 4.____________ ____________ ____________| | | Secure | | User ||ACL/Firewall| | Protocol | __| Access || | | | | | |------------ ------------ | ------------| | || | || \/ || ___________ || | | ||-------> | DUT |<--------|| |-----------Figure 4. Security Configuration ModulePoretsky, Rao, Piatt [ Page 9]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003Measurement units:N/AIssues:NoneSee Also:ACL Configuration SetSecure Protocol Configuration SetPassword Login Configuration Set3.3 Startup ConditionsDefinition:Test conditions that occur at the start of the AcceleratedLife Benchmark to establish conditions for the remainder ofthe test.Discussion:Startup Conditions may cause stress on the DUT and producefailure. Startup Conditions are defined for the ControlPlane and Security Plane.Measurement units:N/AIssues:NoneSee Also:Control Plane Startup ConditionsData Plane Startup ConditionsManagement Plane Startup ConditionsSecurity Plane Startup Conditions3.3.1 Control Plane Startup ConditionsDefinition:Control Plane conditions that occur at the startof the Accelerated Stress Benchmarking to establish conditionsfor the remainder of the test.Discussion:Startup Conditions may cause stress on the DUT and producefailure. Startup Conditions for the Control Plane includesession establishment rate, number of sessions establishedand number of routes learned.Measurement units:N/APoretsky, Rao, Piatt [ Page 10]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003Issues:NoneSee Also:Startup ConditionsSecurity Plane Startup ConditionsControl Plane Configuration Set3.3.2 Security Plane Startup ConditionsDefinition:Security Plane conditions that occur at the startof the Accelerated Stress Benchmarking to establish conditionsfor the remainder of the test.Discussion:Startup Conditions may cause stress on the DUT and producefailure. Startup Conditions for the Security Plane includesession establishment rate, number of sessions establishedand number of policies learned, and number of user accesssessions opened.Measurement units:N/AIssues:NoneSee Also:Startup ConditionsData Plane Startup ConditionsManagement Plane Startup ConditionsSecurity Plane Startup Conditions3.4 Instability ConditionsDefinition:Test conditions that occur during the Accelerated StressBenchmark to produce instability and stress the DUT.Discussion:Instability Conditions are experienced by the DUT after theStartup Conditions have completed. Instability Conditionsoccur for the Control Plane, Data Plane, Management Plane,and Security Plane.Measurement units:N/AIssues:NonePoretsky, Rao, Piatt [ Page 11]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003See Also:Control Plane Instability ConditionsData Plane Instability ConditionsManagement Plane Instability ConditionsSecurity Plane Instability Conditions3.4.1 Control Plane Instability ConditionsDefinition:Control Plane conditions that occur during the Accelerated StressBenchmark to produce instability and stress the DUT.Discussion:Control Plane Instability Conditions are experienced by the DUTafter the Startup Conditions have completed. Control PlaneInstability Conditions experienced by the DUT include sessionloss, route withdrawal, and route cost changes.Measurement units:N/AIssues:NoneSee Also:Instability ConditionsData Plane Instability ConditionsManagement Plane Instability ConditionsSecurity Plane Instability Conditions3.4.2 Data Plane Instability ConditionsDefinition:Data Plane conditions that occur during the Accelerated StressBenchmark to produce instability and stress the DUT.Discussion:Data Plane Instability Conditions are experienced by the DUTafter the Startup Conditions have completed. Data PlaneInstability Conditions experienced by the DUT include interfaceshutdown, link loss, and overloaded links.Measurement units:N/AIssues:NoneSee Also:Instability ConditionsControl Plane Instability ConditionsManagement Plane Instability ConditionsSecurity Plane Instability ConditionsPoretsky, Rao, Piatt [ Page 12]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.4.3 Management Plane Instability ConditionsDefinition:Management Plane conditions that occur during the AcceleratedLife Benchmark to produce instability and stress the DUT.Discussion:Management Plane Instability Conditions are experienced by the DUTafter the Startup Conditions have completed. Management PlaneInstability Conditions experienced by the DUT include repeatedFTP of large files.Measurement units:N/AIssues:NoneSee Also:Instability ConditionsControl Plane Instability ConditionsData Plane Instability ConditionsSecurity Plane Instability Conditions3.4.4 Security Plane Instability ConditionsDefinition:Security Plane conditions that occur during the AcceleratedLife Benchmark to produce instability and stress the DUT.Discussion:Security Plane Instability Conditions are experienced by the DUTafter the Startup Conditions have completed. Security PlaneInstability Conditions experienced by the DUT include sessionloss and policy changes.Measurement units:N/AIssues:NoneSee Also:Instability ConditionsControl Plane Instability ConditionsData Plane Instability ConditionsManagement Plane Instability ConditionsPoretsky, Rao, Piatt [ Page 13]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.5 Failure Conditions3.5.1 Unexpected Session LossDefinition:Session Loss due to Control Plane or Data Plane failure during theAccelerated Stress Benchmarking.Discussion:Session loss will be observed due to Convergence Events producedby the Instability Conditions. This is not considered a failureduring the Accelerated Stress Benchmarking. It is expected behavior.Unexpected packet loss occurs due to Data Plane or Control Planefailure.Measurement units:sessionsIssues:NoneSee Also:Unexpected Packet LossMisrouted PacketsAccess DenialErrored Management ValuesSecurity Plane Failure3.5.2 Unexpected Packet LossDefinition:Packet Loss due to Control Plane or Data Plane failure during theAccelerated Stress Benchmarking.Discussion:Packet loss will be observed due to Convergence Events producedby the Instability Conditions. This is not considered a failureduring the Accelerated Stress Benchmarking. It is expected behavior.Unexpected packet loss occurs due to Data Plane or Control Planefailure.Measurement units:packetsIssues:NoneSee Also:Unexpected Session LossMisrouted PacketsAccess DenialErrored Management ValuesSecurity Plane FailurePoretsky, Rao, Piatt [ Page 14]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.5.3 Misrouted PacketsDefinition:Packets routed to the incorrect next-hop or LSP due to ControlPlane failure during the Accelerated Stress Benchmarking.Discussion:Packets will be rerouted to new next-hops and LSPs due to ConvergenceEvents produced by the Instability Conditions. This is not considereda failure during the Accelerated Stress Benchmarking. It is expectedbehavior. Unexpected Misrouted packets occur due to Control Planefailure.Measurement units:packetsIssues:NoneSee Also:Unexpected Packet LossUnexpected Session LossAccess DenialErrored Management ValuesSecurity Plane Failure3.5.4 Access DenialDefinition:Inability to management the DUT due to the inabilityto gain remote access to the DUT.Discussion:Handling of Instability Conditions may cause a Control Plane failureof the DUT externally observable as the inability to remotely managethe DUT. Whether using SNMP, Telnet, SSH, or FTP, failure to managethe device constitues a failure.Measurement units:Failed sessionIssues:NoneSee Also:Unexpected Packet LossUnexpected Session LossMisrouted PacketsErrored Management ValueSecurity Plane FailurePoretsky, Rao, Piatt [ Page 15]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.5.5 Errored Management ValueDefinition:Incorrect SNMP object value or statistic during the AcceleratedLife Benchmark.Discussion:Management of a router relies upon accurate reporting of objectvalues and statistics. The Accelerated Stress Benchmarkinging validates that the DUT is able to report correct values when experiencingInstability Conditions.Measurement units:Name of incorrect object or statisticIssues:NoneSee Also:Unexpected Packet LossUnexpected Session LossMisrouted PacketsAccess DenialSecurity Plane FailureSuccess Threshold3.5.6 Security Plane FailureDefinition:Violation of Security Policy for packets directed to the DUTor routed through the DUT.Discussion:Security Policy is configured to permit and deny access to packetsdestined for the DUT and packets to be routed through the DUT.A violation of this security policy is when either a packet tobe denied is permitted or a packet to be permitted is denied.Measurement units:packetsIssues:NoneSee Also:Unexpected Packet LossUnexpected Session LossMisrouted PacketsAccess DenialErrored Management ValuePoretsky, Rao, Piatt [ Page 16]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20033.6 Benchmarks3.6.1 Success ThresholdDefinition:The intended number of hours for the DUT to operate without failureduring the Accelerated Stress Benchmarking.Discussion:Success Threshold is a user determined value of time. It isnecessary to define the Success Threshold because it is possiblefor the DUT to experience zero failures for given ConfigurationSets, Startup Conditions, and Instability ConditionsMeasurement units:HoursIssues:NoneSee Also:Accelerated-Life Test Duration3.6.2 Accelerated-Life Test DurationDefinition:The run-time for the Accelerated Stress Test without the DUTexhibiting an error in the control plane, data plane, managementplane, or security plane.Discussion:For a successful test, the Test Duration will equal theSuccess Threshold. In the event of any failure, the Test Durationis less than the Success Threshold. Reporting of the TestDuration MUST inlcude the Failure Condition from those definedin this document.Measurement units:HoursIssues:NoneSee Also:Success ThresholdPoretsky, Rao, Piatt [ Page 17]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20034. Security ConsiderationsDocuments of this type do not directly effect the security ofthe Internet or of corporate networks as long as benchmarkingis not performed on devices or systems connected to operatingnetworks.5. References[1] Bradner, S., Editor, "Benchmarking Terminology for NetworkInterconnection Devices", RFC 1242, July 1991.[2] Mandeville, R., "Benchmarking Terminology for LAN SwitchingDevices", RFC 2285, June 1998.[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology forNetwork Interconnect Devices", RFC 2544, March 1999.[4] "Core Router Evaluation for Higher Availability", ScottPoretsky, NANOG 25, June 8, 2002, Toronto, CA.[5] "Router Stress Testing to Validate Readiness for NetworkDeployment", Scott Poretsky, IEEE CQR 2003.6. Author’s AddressScott PoretskyQuarry Technologies8 New England Executive ParkBurlington, MA 01803USAPhone: + 1 781 395 5090EMail: sporetsky@Shankar Rao950 17th StreetSuite 1900Qwest CommunicationsDenver, CO 80210USAPhone: + 1 303 437 6643Email: srao@Ray PiattCable and Wireless11700 Plaza America DriveReston, VA 20190USAPhone: + 1 703 292 2113Email: rpiatt@Poretsky, Rao, Piatt [ Page 18]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 20037. Full Copyright StatementCopyright (C) The Internet Society (1998). All RightsReserved.This document and translations of it may be copied andfurnished to others, and derivative works that comment on orotherwise explain it or assist in its implementation may beprepared, copied, published and distributed, in whole or inpart, without restriction of any kind, provided that the abovecopyright notice and this paragraph are included on all suchcopies and derivative works. However, this document itself maynot be modified in any way, such as by removing the copyrightnotice or references to the Internet Society or other Internetorganizations, except as needed for the purpose of developingInternet standards in which case the procedures for copyrightsdefined in the Internet Standards process must be followed, oras required to translate it into languages other than English.The limited permissions granted above are perpetual and willnot be revoked by the Internet Society or its successors orassigns. This document and the information contained herein isprovided on an "AS IS" basis and THE INTERNET SOCIETY AND THEINTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTYTHAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANYRIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE.Poretsky, Rao, Piatt [Page 19]INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2003Appendix 1. White Box Benchmarking TerminologyAppendix 1.1 Minimum Available MemoryDefinition:Minimum DUT Available Memory during the duration of theAccelerated Stress Test.Discussion:It is necessary to monitor DUT memory to measure thisbenchmark.Measurement units:bytesIssues:NoneSee Also:Maximum CPU UtilizationAppendix 1.2 Maximum CPU UtilizationDefinition:Maximum DUT CPU utilization during the duration of theAccelerated Stress Test.Discussion:It is necessary to monitor DUT CPU Utilization to measurethis benchmark.Measurement units:%Issues:NoneSee Also:Minimum Available MemoryPoretsky, Rao, Piatt [ Page 20]。