BlueCoat_ISP_Deployment_Best_Practice

Best Practices for ISP Deployment>Table of ContentsNetwork Design and Deployment Recommendations 2 1.1Design goals and assumptions 2 1.2 Cache farm components and design 3 1.2.1 Consumer Edge routers and Policy Based Routing 4 1.2.2 L4-7 Load Balancing Switches 4 1.2.3 L2 Switches 5 1.2.4 Proxy SG 5 1.2.5 Designing for Bandwidth Gain 5 Proof of Concept 6 2.1 PoC 6 Rollout and Service-In 8 Staging Environment and Change Management 9 4.1 Staging environment 9 4.2 Change management 9 Operations 9 5.1 Monitoring 9 5.2 Bypass Lists 10 5.3 Common issues and resolutions 10 Appendices 11 Appendix A. Recommended Settings for Service Providers 11 Trust Destination IP (SGOS version 5.2 or above) 11 HTTP Proxy 11 HTTP Proxy Profile 13 Connection Handling 13 SGOS Feature 17 Appendix B. Tuning Options 17 Investigation and Reporting 17 Tuning 17 Appendix C. Alternative designs 18 C.1 WCCP 18 Appendix D. Vendor Specific Options 20 D.1 F5 20 Overview 20 Cache Load Balancing 21 Optimization options 30IntroductionBlue Coat appliances are deployed primarily as a caching solution in Internet Service Provider environments globally to improve user experience and upstream bandwidth savings. Some ISPs also enable the content filtering option for regulatory or cultural reasons. Deployments cover from small ISP to Tier-1 providers that utilize dozens of Blue Coat ProxySG appliances.This document presents recommended design and deployment best practices. Configuration settings specific to ISPs are explained. Networks that deviate from this best practice may result in less than optimal results in bandwidth savings, stability, or overall performance. Some design alternatives are presented in the appendix along with the caveats of such alternatives.Network Design and Deployment Recommendations1.1 Design goals and assumptionsInternet Service Providers deploy Blue Coat ProxySG devices in order to save upstream bandwidth costs and improve the end-user experience. The goal of this design and deployment best practice is to satisfy these requirements while delivering a scalable and robust cache farm that requires minimal changes to the existing network, complete transparency to end-users and servers, and can be supported with the least possible impact to ongoing operations.In most ISP networks bandwidth gain ranges from 20-30% but is highly dependent on design decisions such as load balancing algorithms, the nature of traffic and web access amongst an ISP’s users, as well as the sizing of the ProxySG devices. Designing for bandwidth gain will be discussed later in this section.The assumption with this best practice design is that the ISP network follows a topology similar to the diagram below. The cache farm should be deployedat a point in the network that is a logical choke point to redirect web traffic into the farm while minimizing the chances of asymmetric routing. Blue Coat’s recommendation is to utilize Policy Based Routing on the core routers that connect the consumer edge (CE) network. Port 80 (HTTP) egress and ingress (bi-directional) traffic is routed from the CE routers to L4-7 load balancing switches that then distribute the HTTP traffic to a “farm” of proxy/caches. The CE routers are recommended because in most networks they provide a choke point without Asymmetric routes. The cache farm can also be tuned to the1.2 Cache farm components and designAs noted above the best practice design utilizes Policy Based Routing to L4-7 load-balancing switches that are connected to a cache farm. The design should include the following:->L4-7 switches should be deployed in a redundant configuration off the core switches/ routers->PBR should be routed to a floating IP/network address on the L4-7 switches for both ingress and egress traffic->L2 switches should be used between the L4-7 switches and the ProxySGs->ProxySGs should be dual homed to the L2 switches->L4-7 switches should be configured with a virtual IP (as in VRRP) to be used as the default gateway for the ProxySGs in the cache farm->L4-7 switches should be configured to use destination based load balancingBased on the ISP topology above the cache farm topology should follow the design shown below. Solid lines represent primary links and dotted lines represent backup links.PBR on consumer edge (CE) routers Edge CE VIP VIP PECoreBGP Peering,IP Core,Internet Exchange 1.2.1 Consumer Edge routers and Policy Based RoutingIn order to eliminate ongoing changes or updates to configuration of core routers the PBR setting should simply forward all port 80 traffic to the VIP of the L4-7 switches. Any bypassing of port 80 traffic should be performed by the L4-7 switches and/or the ProxySGs. PBR will need to be configured both for egress and ingress.In some deployments it may be required to only perform PBR on specificsubnets – for example, only the consumer DSL subnets but not the enterprise customer subnets. Best practice is to perform PBR on the CE routers which would avoid this issue since each set of CE routers would already be dedicated to the different types of customers.Important: Locally hosted Akamai servers or similar internal CDNinfrastructure should be excluded from the PBR.Configuration notes for specific routers may be found in the Appendix.1.2.2 L4-7 Load Balancing SwitchesThe L4-7 switches should be capable of load balancing based on destination. Optimal caching performance will be obtained if destination is URI orcomponent of URI instead of destination IP.L4-7 switch load balancing must be flow based instead of packet based.Packets forwarded from the L4-7 switches must include the client IP address.Proxies should use the L4-7 switches as default gateway. L4-7 switch must be capable of passing the return traffic to the correct proxy.Configuration notes for specific L4-7 switches may be found in the Appendix.1.2.3 L2 SwitchesL2 switches are used between the L4-7 switches and Proxy SGs to provide the active-standby connectivity, future expansion and cost savings. Spanning tree should not be necessary on the L2 switches and not recommended due to the extra overhead. This implies of course that the L2 switches are not directly interconnected but that the L4-7 switches and Proxy SGs are dual-homed to each L2 switch.1.2.4 Proxy SGSGOS versionBlue Coat recommends that all service providers run SGOS4.2 (the current release as of 10/1/2008 is SGOS4.2.8.6).ISP specific configurationThere are some configuration options of the Proxy SG recommended for ISPs that veer from ProxySG default settings. Please refer to the Appendices for an detailed review of the recommended changes.1.2.5 Designing for Bandwidth GainCaching is a statistics game. The Internet contains billions and billions of objects. Each proxy can only cache a finite amount of content; in order to provide optimal performance and bandwidth savings the most popular content must be effectively distributed across a largest number proxies possible.Cache performance is a function of total number of disk spindles, memory, and CPU. The bandwidth savings of the entire cache farm is tied to total number of unique web objects the farm can cache, which in turn is tied to the total disk spindles. Memory effects both how much data can be cached in RAM as well as how many connections can be simultaneously handled. CPU dictates the total throughput of an individual proxy. Sizing should be done to satisfy throughput requirements as well as caching requirements. The solution must be capable of handling peak HTTP load, and ideally there should be enough disk space to store the equivalent of 2-3 days of HTTP traffic.Caching StrategyThe load balancing solution should be capable of load balancing basedon destination, ideally based on components of the URL rather than by IP destination. This prevents duplicate content from being distributed across the caching layer. Additionally, the load balancing layer can be used to implement and refine how the caches are deployed to maximize bandwidth gain. Refer to the tuning section of the Appendices for more details.Platform SelectionBased on the above requirements there are two Blue Coat models that make sense for service providers, the 810-25 and 8100-20. Each has different pros and cons:8100-20• Pros˚8 disks in a single 4U chassis˚4 CPU cores allow 2-3x the throughput of the 810-25• Cons˚Half the disk density of the 810-25 – 8 disks in 4U vs. 16810-25• Pros˚Allows for the greatest disk density – each system provides 4 spindles ina single rack unit˚2 CPU cores allow peak throughput of 100-150 Mbps• Cons˚Single Power supply˚More systems to manage as compared to 8100 platformProof of Concept2.1 PoCProof of Concept testing should focus on validation of interoperability, functionality, basic management, and troubleshooting. Performance testing should be out of scope for Proof of Concept testing. Information regarding QA and load testing done for Blue Coat’s products can be provided under NDA. Due to the nature of testing Blue Coat functionality and features in an ISP environment it is impractical to attempt to do any sort of simulated testing.It’s simply not possible to generate traffic which simulates real users andreal internet servers. A simulated test can only give a general sense of how much load a platform might handle. Bandwidth gain and precise performance measurements could never be evaluated. Additionally, every ISP has slightly different environments, routers, switches, L4-7 redirection gear. Generally speaking this equipment is very expensive equipment that cannot be easily or effectively duplicated in a lab. Blue Coat regularly does simulated tests internally and information on such tests can be provided to a customer as needed. However, if part of the POC criteria requires testing under load, Blue Coat strongly recommends using live traffic. The POC can follow the rough outline illustrated here.Lab TestingIn the event that a lab environment is maintained with identical equipment to the production routing and switching environment then some basic lab integration testing is recommended. The intent of this testing is to make sure that all the pieces of equipment work together with the recommended software revisions. This step is only useful if identical equipment exists in the lab, testing on smaller versions of the same type of equipment will typically lead to misleading results. It can be done, but the experience learned from that equipment should be interpreted with caution; it should not be assumed that lessons learned on lower classes of equipment apply directly to the high end.Initial installationThe proxy should be installed into its final resting place in the network. Because traffic must be redirected to the proxy it can do no harm in this location in the network. At this point basic connectivity to the internet and client networks should be established. Connections to the management network should also be set up at this time.Explicit Proxy TestingThe proxy should be configured with a rule set to deny all traffic except specific source subnets. Explicit proxy should then be enabled. Basic load testing should be done to ensure that the proxy has adequate connectivity to DNS, default and backup routes, etc. Blue Coat typically recommends Web Timer as a simple tool to do this sort of testing. This can also be used to illustrate performance gains and bandwidth savings with a small group of web sites.L4-7 testingTesting basic L4-7 functionality will have some minimal impact on the production network; the degree of impact depends on the topology chosen. As a best practice Blue Coat recommends using policy based routing functionality to route traffic to the L4-7 switching devices. At this point a specific test network or set of IP addresses should be identified. In this case a test network means a network without any real user traffic on it, solely testing workstations.A policy based route must be added to the production routers to redirect port80 traffic to the test network to the L4-7 device.This should generally be configured in stages:1st stage redirects traffic with a single PBR for outbound port 80 traffic. IP spoofing is not enabled on ProxySG. Basic L4-7 functionality and load balancing is then tested, things like failover can also be tested at this stage.2nd stage adds a second PBR for source port 80 return traffic. This tests to make sure the L4-7 device is returning the L4-7 traffic to the correct Blue Coat device and that there are no asymmetric routes.End to End Functionality testingAt this stage the customer should test from their test network all the basic functionalities expected from Blue Coat.Customer testing:Identify specific customer subnets – redirect them, if desired during off-peak hours. Monitor via access log, statistics, and actual end user experience from a real client if possible.Rollout and Service-InIf the PoC was conducted on the production network the service-in is straightforward as the equipment is already in place.Deploying into the live network follows the same steps recommended above for Proof of Concept.During the PoC or the rollout it is recommended that operations and support procedures be validated. Common items to test are:-> Software upgrade and downgrade/rollback-> Configuration changes (see the Staging Environment and Change Management-> Troubleshoot and resolve a « problem website » using bypass lists (see the Operations section below)-> SNMP traps for CPU load and memory pressure on the Proxy SGs-> Failover and failback scenarios in the overall caching infrastructure (load balancer outage, proxy outage, etc.)Staging Environment and Change Management4.1 Staging environmentBlue Coat highly recommends that all service provider solution also have a staging environment. This staging environment does not necessarily needto replicate the scale and completeness of the production network but canbe a smaller Proxy SG where configuration changes can be validated before deploying into the live network.The Proxy SG used for staging can be:-> standalone in a lab where a client PC is placed on the LAN-side port and transparently proxied to the Internet with the test configuration-> deployed in the live network but only available to the tester using explicit proxy-> deployed in the live network along side the production caches and the test host or network can be explicitly load balanced to it4.2 Change managementBlue Coat highly recommends a strict change management policy combined with the above staging environment to ensure minimal disruption to the production network.Change management should encompass:-> SGOS version upgrades-> Proxy SG configuration changes-> Bypass list changes/updates-> Load balancing configuration changesOperations5.1 MonitoringProxySG comes with industry standard SNMP v1, v2 and v3 support. Private MIBs are available for download in BlueCoat website. Monitoring of ProxySG is fully described in document “Critical Resource Monitoring of the ProxySG”.5.2 Bypass Lists5.3 Common issues and resolutionsContent RefreshThe Adaptive Refresh algorithm within ProxySG is the only technology in the industry that develops a “Model of change” for every Web object in its store.It also develops a “Model of use” based upon that object’s history of being requested by users. It then combines these two pieces of information to determine the refresh pattern appropriate to that object. (The values derived by the algorithms also adapt to changes produced by these Models over time.)Using the Adaptive Refresh algorithm the accelerator automatically performs “freshness checks” with the origin server to ensure that old content is expunged and replaced with fresh content. For example, if the objects within the home page are popular among the population of users that are accessing the accelerator, ProxySG will update the objects that change (e.g., “top story” object) but will not refresh those objects that do not change (e.g., “NBC logo” object). This ensures that current content will be delivered to end users quickly.If an end user does complain about stale content is served from certain websites, ISP administrator can put website URLs onto local bypass list. The down side of it is ProxySG will not cache content from those websites. Another way to handle this situation is to develop CPL to verify each request to these sites. If content is same as the cached one, ProxySG serves the cached content. If not, the latest content is pulled from the website.Sample CPL to verify content:; ---------------start here-----------------define condition CHECK-HOSTSurl.domain=url.domain=url.domain=end<cache> condition=CHECK-HOSTSalways_verify(yes); ---------------start here-----------------Note: Use “Always Verify” only for problematic websitesAppendicesAppendix A. Recommended Settings for Service ProvidersTrust Destination IP (SGOS version 5.2 or above)This is a feature to trust the client provided ip address provided in HTTP request and not to perform DNS lookup on it. This will have impact on cache store performance as it IP address is used as reference instead of domain name. It means same web content in load-balanced website with multiple IP addresses can be stored multiple times as it is considered different objects. Cache hit ratio can also be affected for ISP’s customers that are used to input IP address in browser instead of proper domain URL.In the case where the load balancing solution is packet based (not flow based) such as WCCP, this feature is required.Best Practice:-> Do not turn on unless it is required.HTTP Proxyalways-verify-source– Ensures that every object is always fresh upon access. This has a significant impact on performance because HTTP proxy revalidates requested cached objects with the OCS before serving them to the client, resulting in a negative impact on response times and bandwidth gain. Therefore, do not enable this configuration item unless absolutely required.max-cache-size– The maximum size, in megabytes, of the largest object that can stored on the ProxySG. The max-cache-size sets the maximum object size for both HTTP and FTP.Refresh Bandwidth– The ProxySG uses as much bandwidth as necessary for refreshing to achieve the desired access freshness – 99.9% estimated freshness of the next access. The amount of bandwidth used varies depending on client demands. If you determine that the ProxySG is using too much bandwidth (by reviewing the logged statistics and examining current bandwidth used shown in the Refresh bandwidth field), you can specify a limit to the amount of bandwidth the ProxySG uses to try to achieve the desired freshness.This setting should only be changed from default (Let the ProxySG Appliance manage refresh bandwidth) to limit bandwidth to if ISP upstream bandwidth is at its premium.Best Practice• Always-verify-source should be disabled (default)• For SG8100 model that uses 300G hard drive. Change Set Max-cache-size to 100MB. This allows more objects to be cached in each hard disk.• For SG8000 model that uses 37G hard drive. Change Set Max-cache-size to 10MB.• Use default for Refresh Bandwidth but modify if necessaryPragma-no-cache– The pragma-no-cache (PNC) header in a client’s request can affect the efficiency of the proxy from a bandwidth gain perspective. If you do not want to completely ignore PNC in client requests (which you can doby using the Substitute Get for PNC configuration), you can lower the impactof the PNC by enabling the revalidate-pragma-no-cache setting. When the revalidate-pragma-no-cache setting is enabled, a client’s non-conditional PNC-GET request results in a conditional GET request sent to the OCS if the object is already in the cache. This gives the OCS a chance to return the 304 Not Modified response, thus consuming less server-side bandwidth, becauseit has not been forced to return full content even though the contents have not actually changed. By default, the revalidate PNC configuration is disabled and is not affected by changes in the top-level profile. When the Substitute Get for PNC configuration is enabled, the revalidate PNC configuration has no effect. Note: The revalidate pragma-no-cache setting can only be configured through the CLI.Best Practice-> Enable revalidate-pragma-no-cache from CLIByte Range Support– With byte-range support enabled, if the object is already cached and does not need to be reloaded from the OCS, the ProxySG serves the byte-range request from the cache only. But if the object is not in the cache,or if a reload of the object is required, the ProxySG might treat the byte-range request as if byte-range support is disabled and serve the object from the cache. If byte-range support is disabled, HTTP treats all byte-range requests as non-cacheable. Such requests are never served from the cache, even if the object exists in the cache. The client’s request is sent unaltered to the OCS and the response is not cached. Thus a byte-range request has no effect on the cache if byte-range support is disabled.Best Practice-> Enable byte range support which is defaultHTTP Proxy ProfileThree preset profiles, Normal, Portal, and Bandwidth Saving, are selectable from SG. Each profile includes different settings on HTTP proxy behavior. Bandwidth Saving profile is recommended for ISP environment.Best Practice-> Enable Bandwidth Saving Profile-> Disable Substitute GET for PNC (Pragma no cache)-> Disable Substitute GET for IE (Internet Explorer) reload----------------------------------------------------------------------------------Connection HandlingBypass ListsA bypass list can be used to completely skip all ProxySG processing of requests sent to specific destination hosts or subnets. This prevents the appliancefrom enforcing any policy on these requests and disables any caching of the corresponding responses, so it should be used with care. A bypass list allows traffic to pass through to sites as-is when servers at the site are not properly adhering to protocol standards or when the processing in the ProxySG is otherwise causing problems. The bypass list contains IP addresses, subnet masks, and gateways. When a request matches an IP address and subnet mask specification in the bypass list, the request is sent to the designated gateway and is not processed by the ProxySG.Three types of bypass lists are available:-> Local-> Central-> DynamicLocal Bypass ListThe gateways specified in the bypass list must be on the same subnet as the ProxySG. The local bypass list limit is 10,000 entries. The local bypass list contains a list of IP addresses, subnet masks, and gateways. It can also define the default bypass gateway to be used by both the local bypass list and central bypass list. The gateways specified in the bypass list must be on the same subnet as the ProxySG. When you download a bypass list, the list is stored in the appliance until it is replaced by downloading a new list.Central Bypass ListThe central bypass list is usually a shared list of addresses that is used by multiple ProxySG Appliances. Because each ProxySG appliance can be located on a different subnet and can be using different gateways, the central bypass list should not contain any gateway addresses.The gateway used for matches in the central bypass list is the gateway specified by the bypass_gateway command in the local bypass list. If there is no bypass_ gateway option, the ProxySG uses the default gateway defined by the network configuration.Dynamic BypassDynamic bypass, available through policy (VPM or CPL), can automatically compile a list of requested URLs that return various kinds of errors. The policy-based bypass list is maintained in the Forward Policy file or Local Policy file. A bypass list can be used to completely skip all ProxySG processing of requests sent to specific destination hosts or subnets. This prevents the appliancefrom enforcing any policy on these requests and disables any caching of the corresponding responses, so it should be used with care. A bypass list allows traffic to pass through to sites as-is when servers at the site are not properly adhering to protocol standards or when the processing in the ProxySG is otherwise causing problems.Dynamic bypass is most useful in ISP environment. It keeps its own (dynamic) list of which connections to bypass, where connections are identified by both source and destination rather than just destination. Dynamic bypass can be based on any combination of policy triggers. In addition, some global settings in HTTP configuration can be used to selectively enable dynamic bypass based on specific HTTP response codes. Once an entry exists in the dynamic bypass table for a specific source/destination IP pair, all connections from that source IP to that destination IP are bypassed in the same way as connections that match against the static bypass lists.With dynamic bypass, the ProxySG adds dynamic bypass entries containing the specific source/destination IP pair for sites that have returned an error to the appliance’s local bypass list. For a configured period of time, further requests for the error-causing URLs are sent immediately to the origin content server (OCS), saving the ProxySG processing time. The amount of time a dynamicbypass entry stays in the list and the types of errors that cause the ProxySG to add a site to the list, as well as several other settings, are configurable from the CLI. Please refer to Blue Coat Configuration and Management Guide for detail.Once the dynamic bypass timeout for a URL has ended, the ProxySG removes the URL from the bypass list. On the next client request for the URL, the ProxySG attempts to contact the OCS. If the OCS still returns an error, the URL is once again added to the local bypass list for the configured dynamic bypass timeout. If the URL does not return an error, the request is handled in the normal manner.Default SettingDynamic Bypass is disabledserver_bypass_threshold = 16max_dynamic_bypass_entry = 16,000dynamic_timeout = 60Best Practice-> Enable dynamic bypass with trigger for connection and receiving errors.-> Adjust default settings to suit network condition.-> Individual HTTP response code(e.g 404 ) can be configured as trigger if necessary. HTTP TimeoutYou can configure various network receive timeout settings for HTTP transactions. You can also configure the maximum time that the HTTP proxy waits before reusing a client-side or server-side persistent connection. You must use the CLI to configure these settings. Default HTTP Receive Timeout for client is 120 seconds, for server is 180 seconds. Default HTTP Persistent Connection Timeout for client is 360 seconds server and for server is 900 seconds. It is recommended to lower the timeout values in ISP environment so that ProxySG can reuse its client/server workers effectively.Best Practice-> http persistent-timeout server 20-> http persistent-timeout client 10Error HandlingDefault proxy behavior when there are TCP timeouts or DNS failures is to display an error page generated by the proxy. In an ISP environment it is preferred to instead terminate the connection and let the end user’s browser。