Copyright 2015, Synopsys, Inc. All rights reserved worldwide.
Table of Contents
1. Introduction (1)
1.1. Prerequisites (1)
1.2. Requirements (1)
1.3. How to Use This Guide (2)
1.4. A Wealth of Reference Information (2)
1.5. The Pirate Code of Fuzzing (2)
1.5.1. Don’t fuzz production targets. (2)
1.5.2. Don’t fuzz targets that will be in production. (2)
1.5.3. Get physically close to your target. (2)
1.5.4. Get logically close to your target. (3)
1.6. Attack Surface Analysis (3)
1.7. Getting Help (4)
2. Running Defensics for the First Time (5)
2.1. Configuring the Suite Directory (5)
2.2. Connect to the License Server (6)
2.3. Connect to the Arena (8)
3. Quick Start (11)
3.1. Choose a Target (11)
3.2. Load a Test Suite (11)
3.3. Learn One, Learn Them All (11)
3.4. Basic Configuration (12)
3.5. Interoperability (13)
3.6. Test Run (15)
3.7. Using Testplans and Settings (16)
3.8. About Benchmark (16)
3.9. Congratulations! (17)
4. Interoperability (18)
4.1. Check the Defensics Main Log (18)
4.2. Connectivity Problems (20)
4.3. Configuration Problems (20)
5. Results (22)
5.1. Overview (22)
5.2. Top Level Actions (23)
5.3. Statistics (23)
5.4. The Main Log (25)
5.5. Viewing Live Results (27)
5.6. Where Are the Files? (27)
5.7. Results Notes (27)
5.8. Notes Templates (29)
6. Test Cases (30)
6.1. Test Case Selection (30)
6.1.1. The Tree View (31)
6.2. The List View (32)
6.3. The Test Case Selection Field (33)
6.4. The Test Cases Button (33)
6.4.1. Controlling Test Case Generation (33)
ii
6.4.2. Limited Test Case Selection Modes (34)
6.5. Looping and Repeating (34)
6.6. About Test Case Indexes (34)
7. Instrumentation (36)
7.1. What is Failure? (36)
7.2. Valid Case Instrumention (37)
7.2.1. Enabling Valid Case Instrumentation (37)
7.2.2. Adjusting the Timeout (38)
7.2.3. Multiple Rounds of Instrumentation (38)
7.2.4. Instrumentation Fail Limit (38)
7.3. TCP Instrumentation (38)
7.4. Syslog Instrumentation (39)
7.5. SNMP Trap Instrumentation (40)
7.6. SNMP Query Instrumentation (40)
7.6.1. Understanding SNMP Query Instrumentation (40)
7.6.2. Using the SNMP Scanner (41)
7.6.3. Setting up SNMP Instrumentation (43)
7.6.4. Examining SNMP Instrumentation Results (45)
7.7. External Instrumentation (47)
7.7.1. Ping Example (47)
7.7.2. What Can Defensics Call? (48)
7.7.3. Other Hooks (48)
7.8. Eyeballs (48)
8. Remediation and Reports (50)
8.1. Reliable Failure (50)
8.2. Remediation Workflow (50)
8.3. Creating a Remediation Package (50)
8.4. How a Developer Uses a Remediation Package (53)
8.5. It Works Even When It’s Not Your Software (54)
8.6. Generating Reports (55)
9. Bug Hunting (57)
9.1. Did I Find a Bug? (57)
9.2. Boom! (57)
9.3. That’s Nice, But Can You Do It Again? (59)
9.4. Dig Deeper (60)
9.5. Steppin' Out With My Test Cases (61)
9.6. Looping and Repeating (62)
10. Attack Surface Analysis (64)
10.1. Common Attack Vectors (64)
10.2. Make a List, Check it Twice (64)
10.3. Detective Work (65)
11. Automated Fuzzing (67)
11.1. First Things First (67)
11.2. The Simplest Method: Use the Command Line (67)
11.3. If You Prefer, Use the HTTP API Instead (68)
11.4. Now That You Got Your Feet Wet, RTFM (69)
12. Working with Message Sequences (70)
12.1. Tour of Message Sequences in SIP UAS (70)
iii
12.2. Editing Message Sequences (71)
12.3. Editing Message Content (72)
12.4. With Great Power Comes Great Responsibility (76)
13. Performance (77)
13.1. Optimizing the Response Timeout (77)
13.2. Tuning Instrumentation Settings (79)
13.2.1. Choose the Valid Case Wisely (79)
13.2.2. Tuning the Valid Case Instrumentation Frequency (80)
13.2.3. Optimize External Instrumentation (80)
13.3. Specific Settings for Windows (81)
13.3.1. Source Port Ranges (81)
13.3.2. TCP Wait Timeout (81)
14. Parallelization (82)
14.1. Chaos Parallelization (82)
14.2. Dandelion Parallelization (82)
14.3. Using Test Suite Settings (82)
14.4. Using Testplans (83)
14.5. Fuzzing Farms (83)
15. Client and Passthrough Fuzzing (84)
15.1. Client Testing (84)
15.1.1. You’re Going to Need Automation (84)
15.1.2. An Example Based on curl (84)
15.2. Passthrough Testing (85)
16. File Fuzzing (87)
16.1. Selecting a Delivery Option (87)
16.2. Export Test Cases as Files (87)
16.3. Execute Command (88)
16.4. TCP Connection (88)
16.5. UDP Datagram (88)
16.6. HTTP Server (88)
iv
Chapter 1. Introduction
This document describes how to use Defensics for fuzz testing and some general guidelines for safe testing.
1.1. Prerequisites
If you don’t already understand fuzz testing, or if you want to refresh your memory, try these resources:?Read: What is Fuzzing? The Poet, the Courier, and the Oracle (PDF, 11 pages)
?https://www.doczj.com/doc/eb3009205.html,/resources/white-paper/2015/01/20/what-is-fuzzing.html
?Read: Make Software Better with Fuzzing (PDF, 5 pages)
?https://https://www.doczj.com/doc/eb3009205.html,/file/d/0B5kHdWiepFoIYkdOeE0zbUdxd28/view?usp=sharing
?Watch: Training videos
?Unknown Vulnerability Management and Discovery Using Fuzzing
?https://https://www.doczj.com/doc/eb3009205.html,/album/2805910/video/88662183
?Part 1: What are unknown vulnerabilities and why should I care
?Part 2: What is fuzz testing, and where does it fit in the world of software?
?https://https://www.doczj.com/doc/eb3009205.html,/album/2805910/video/89234167
?Part 3: How and why fuzz testing, and managing your unknown vulnerabilities saves money
?https://https://www.doczj.com/doc/eb3009205.html,/album/2805910/video/89974203
?Part 4: Fuzz testing techniques: unfuzzing the fuzzing
?https://https://www.doczj.com/doc/eb3009205.html,/album/2805910/video/90074270
1.2. Requirements
You’ll learn best if you work with Defensics as you’re reading this guide. Try things as you go along! Practice with Defensics to improve your skills.
Before you start reading the rest of this guide, make sure you have Defensics installed. Consult the Defensics Installation Guide for details. Setup steps for running Defensics for the first time are in the next chapter.
Remember, software testing is a noble pursuit. You are finding bugs so that they can be fixed, making the world safer, more robust, and more secure. Keep honing your skills to find as many bugs as you can. It’s your job to break things!
1
1.3. How to Use This Guide
You should read straight from the beginning of this guide to the chapter on remediation. This will give you the full life cycle of using Defensics to locate and fix bugs in software.
The rest of the chapters are recommended, but you can choose topics as necessary if you’re short on time. Not everyone, for example, will have to tackle file testing. Feel free to read chapters as you need them, and in any order.
This chapter describes the larger picture of fuzz testing. You’ll read about general guidelines for fuzz testing as well as how to approach testing a specific target.
1.4. A Wealth of Reference Information
Defensics contains copious help text inside the application itself. Click on any field for context-sensitive help in the right pane.
For more general information, choose GUI Help or Suite Help in the right pane.
The purpose of this document is to supplement the reference information available in Defensics with clear explanations of the process of fuzzing with Defensics.
1.5. The Pirate Code of Fuzzing
A good fuzzer is dangerous. Defensics is very dangerous. The whole point of fuzz testing is to break things—this is how you locate vulnerabilities so that they can be fixed.
With this in mind, a few simple guidelines will keep you from getting in serious trouble. These are general guidelines rather than strict rules.
1.5.1. Don’t fuzz production targets.
Fuzzing is destructive testing and will be perceived as an act of aggression. Don’t test anything that anyone is using.
1.5.
2. Don’t fuzz targets that will be in production.
Fuzz testing can leave targets in unusual internal states. For example, a device that still appears to respond normally after testing might have a severely damaged internal database.
If you were a patient in a hospital, would you want to use that infusion pump that had previously been fuzz tested?
If you are confident that the tested device can be returned to its factory state, you might be able to use it after fuzz testing. Use your common sense.
1.5.3. Get physically close to your target.
Any devices or equipment between Defensics and the test target can cause three kinds of trouble.
2
1.The testing speed you are able to achieve will be lower if the test cases have farther to travel.
2.Devices between Defensics and the target might modify or discard test cases before they reach the
target.
3.Test cases sent by Defensics to the target might end up causing a failure in one of the intermediary
hops. For example, a firewall that parses and examines protocol messages as they pass through might get broken by a test case from Defensics.
Let’s say, for example, that you are fuzzing SIP (Session Initiation Protocol) on a piece of VoIP (Voice over IP) equipment.
Wrong: Install Defensics on your desktop computer in your cubicle. Point the SIP fuzzer at the target in the lab and fire away!
This is a bad idea. Let’s say your desktop computer is connected to a switch, which connects through
a firewall to the la
b switch, which is connected to the target device. If either of switches examines SIP messages (perhaps for optimization), or the firewall examines SIP messages, you might end up breaking one of the switches or the firewall instead of your target.
Even if the switches and the firewall don’t fail, they might modify or discard test cases before they reach the target.
Even if none of that happens, you’ll be flooding the office and lab network with traffic from Defensics. Somebody will be angry about it.
Much better: Install Defensics on a machine in the lab and connect through the lab switch to the target. Alternately, install Defensics on a laptop and connect through the lab switch to the target.
Best: Plug your Defensics machine directly into the target device.
1.5.4. Get logically close to your target.
Some targets have built-in security countermeasures, such as firewalls and connection throttling.
Our recommendation is to disable those features to make the target as vulnerable as possible. The point of fuzzing is to find and fix vulnerabilities. Disabling security countermeasures means you will find the maximum number of bugs given the tools and time you have available.
On the other hand, disabling these features means that you are testing a target that is not in its eventual production configuration. You’ll have to decide what’s most important to you. Perhaps you can perform most of your testing with security features disabled. After fixing all the bugs you find, re-enable the security features and re-test. If you’re short on time, the second set of testing could be a shorter test run.
1.6. Attack Surface Analysis
You need to understand a target’s attack surface to do meaningful security testing. The attack surface is a collection of attack vectors, where an attack vector is a place that the target takes any form of input. Attack surface analysis is described more fully elsewhere, including the Fuzz Testing Maturity Model. For now, think of it as a list of protocols that are supported by the target.
3
Based on your list of attack vectors, you can plan the fuzz testing that you intend to do. Which test suites will you use? What configurations? How much testing will you do? Again, the Fuzz Testing Maturity Model can help you answer these questions.
1.7. Getting Help
Defensics is supported by a global team of security engineers. Ask for help in any of the following situations:
?You need help installing Defensics or installing a license server.
?You are having trouble with interoperability.
?You’ve located a bug in Defensics or a test suite.
?You want to request an enhancement to Defensics or a test suite.
Contact Defensics support by sending an email to support@https://www.doczj.com/doc/eb3009205.html,, and cc anyone you might know, such as a local engineer.
For the quickest and best response, provide the following information with your question:
?Test results as needed
?Packet captures as needed
?Configuration files
?Anything else that might help
Defensics helps you package information for support. Use one of the following choices from the Defensics menu:
?Help > Report Defensics issue > A bug in test suite or user interface
?Help > Report Defensics issue > Test suite valid case problem
4
5
Chapter 2. Running Defensics for the First Time
The first time you run Defensics, you need to configure a few things:
1.Tell Defensics where to store test suites.
2.Connect to the license server.
3.Connect to the Arena and install one or more test suites.
There are three prerequisites:
1.You should already have installed Defensics. Consult the Defensics Installation Guide for instructions.
2.You need to know the IP address and port number for the license server. Consult the license server administrator at your organization to get this information.
3.You should have a set of Arena credentials. You can request credentials from support@https://www.doczj.com/doc/eb3009205.html, .
2.1. Configuring the Suite Directory
When you first run Defensics, it looks like this:
6
The left side of the window shows recently opened test suites and test plans. It’s empty right now. If your test machine is connected to the Internet, the right side shows the latest Defensics news, including recent releases.
To get started, open the test suite browser by clicking the big Open suite browser button.
Defensics will prompt you to choose an installation directory for test suites. It gives you a default value which is the same as the location where the Defensics application is installed. I like to choose a suites
subdirectory, but you can choose any location you wish.
Click OK .
2.2. Connect to the License Server
Next, Defensics notices that it is not hooked up to a license server.
Click Yes . Defensics opens the License Manager , which has an empty list of licenses.
7
Click Set server address , then choose the appropriate option. For example, if you are connecting to a license server elsewhere in your organization, chooose the third option and fill in the IP address and port number of the license server. Here is an example for a license server located at 10.0.1.199 on the default
port (27000):
Click OK . The License Manager now shows a list of licenses retrieved from the server.
That’s all you need to do. Next time you run Defensics, it will remember the location of your license server and connect automatically.
When you click Close in the License Manager, Defensics presents you with an End User License Agreement. Read it carefully and click I accept if you are satisfied with the terms.
2.3. Connect to the Arena
Defensics remembers that you were originally trying to load a test suite and automatically opens the Suite Browser. Because you’ve just installed Defensics, there are no test suites available to load.
8
To install one or more test suites, click Download. Defensics shows you a list of the test suites for which you have licenses.
Check off one or more suites in the list. Defensics prompts you for your Arena credentials. Enter your
credentials, check off Save Arena password, and click Login.
9
Now click Download and install. Defensics retrieves the selected test suites from the Arena and installs them. Click Close to dismiss the download window.
The Suite browser now shows the test suites you just installed.
You have now completed the first-time setup of Defensics. Next time you run Defensics, you’ll be able to
simply load a test suite and start testing.
10
Chapter 3. Quick Start
One of Defensics' strengths is its user interface. Like any good user interface, it makes easy tasks easy and difficult tasks possible. All test suites share a common user interface, so once you’ve learned how to use one, you can use them all.
In this chapter you’ll learn how to configure Defensics and start testing. Many of the details will be explained fully later.
3.1. Choose a Target
If you want to work through the examples in this document as you’re reading (which is highly recommended), you’ll need a test target. Make sure you pick something that is not in production, that nobody cares about.
A virtual machine is an excellent choice for target practice. I suggest starting with some kind of HTTP server, because the HTTP protocol is relatively simple, human-readable, and easy to verify with a web browser. You might, for example, create a Linux virtual machine and run Apache or nginx as an HTTP server.
Discarded Wifi access points, printers, or other office equipment often includes a web interface, so you could use actual physical old equipment for HTTP target practice as well.
Use your imagination, but be careful about what you’re testing and the network upon which you test. A direct connection from your test machine to the target is safest.
3.2. Load a Test Suite
To load a test suite, open the Suite browser. There are actually four ways to get there:
1.Click on the big Open suite browser button. Note that this button is not always visible.
2.
Click on the suite browser button in the toolbar.
3.Choose File > Open suite browser from the menu.
4.Choose Suites > Load suite from the menu.
Choose the suite you want (HTTP Server in our example) and click Load.
Note that the newly loaded test suite is in its own tab. You can load multiple test suites, each of which operates independently. Don’t get carried away, though, because the number of test suite instances you can comfortably run depends on your available memory.
3.3. Learn One, Learn Them All
Defensics lays out the workflow of finding and reporting bugs in eight steps, which you can see in the left side of the suite window. You can click on each step and see its details in the middle pane. You can click on any field in the middle pane to see help text in the right pane.
11
12
In the example here, you are looking at 4) Instrumentation and the right pane is showing help about the Instrumentation fail limit
.
One of the strengths of Defensics is that this same user interface is used for every test suite. Once you learn how to use one test suite, you know how to use them all.
In this chapter, you’ll really only worry about steps 1, 2, and 6.
3.4. Basic Configuration
The first step in any fuzz testing is to aim the test suite at the target. For many protocols, you just specify an IP address and a port number.
In this HTTP Server test suite, you’ll use a URI to tell Defensics where to find the target. This is just like what you would need to type into your browser to load a page.
Click on 1) Basic configuration to enter the URI.
In my test lab, I am running a web server on port 80 at IP address 192.168.56.102, which looks like this:
As you can see, 1) Basic configuration contains a handful of substeps, such as HTTP authentication, Proxy, and so forth. The exact details depend on the protocol being tested—the IPsec test suite, for instance, will have options that pertain to the IPsec protocol.
Fill these in if necessary. For example, if you are trying to access an authenticated web page, fill in the credentials in the HTTP authentication substep.
Many times, the default values are sufficient.
3.5. Interoperability
The second step is always interoperability testing. Again, this applies no matter which test suite you are using.
Click on 2) Interoperability and click Test. Defensics doesn’t do any fuzzing, but instead exchanges valid messages with the target. This serves two purposes.
1.Defensics is checking to make sure everything is hooked up correctly. It sends valid messages to the
target (in this case, a valid HTTP request) and looks for valid responses (an HTTP response from the target).
2.Defensics is figuring out which protocol features are supported by the target. Many protocols have
optional components, such as messages that the target might or might not recognize. It doesn’t make sense to fuzz test features that aren’t supported, so Defensics uses a series of valid cases to test for various features.
HTTP does not have this kind of structure, so there is just one valid case that gets tested for interoperability.
If everything works according to plan, the valid case turns green, indicating that Defensics was able to successfully exchange valid messages with the target.
13
If the valid case turns red instead, try clicking on Log to see the error Defensics is producing. Double-check your configuration. You can use a web browser to try to pull up the target web page as well.
Be aware that some protocols will have multiple valid cases, and your target might simply not supoprt them all.
Some protocols have optional parts, and part of the purpose of Defensics' interoperability testing is to determine which parts of the protocol are supported by the target. There is no point in fuzzing message types that aren’t understood by the target anyhow.
Aside from green and red, you might see yellow during interoperability testing. This is an indication that Defensics was unable to determine the success or failure of the valid messaging.
For example, if Defensics sends a valid message but the protocol being tested does not dictate a mandatory response, Defensics has no way of determining if the target recognized the message. In this case, the valid case will turn yellow.
For more details on getting interoperability to succeed, consult the next chapter.
Blue check marks on the valid cases indicate which are supported by the target. The test cases corresponding to the check marks will be generated. By default, Defensics puts a blue check mark on each green and yellow valid case. You can adjust this configuration manually if you wish, enabling or
disabling corresponding groups of test cases.
14
15
3.6. Test Run
Once interoperability succeeds, you can begin testing. Click on 6) Test run
and click on the "play" button.Defensics begins delivering test cases to the target. The heartbeat display shows a spike every time a test case is delivered to the target. Spend some time looking at this window while the testing is running.Defensics shows you a variety of interesting information:
?The current test case
?The total number of test cases in this test run
?The average speed of delivering test cases to the target, both for the last minute and for the entire test run
?The elapsed time of this test run
?An estimate of the remaining amount of time for this test run
The estimate of remaining time is just a guess based on the performance of Defensics and the target.Many factors affect testing speed; the target could slow down over time, or later test cases might take longer than earlier ones (especially for multi-message sequences), so don’t be surprised if the testing actually takes longer than the estimated time.
3.7. Using Testplans and Settings
Once you get things configured just right, you can save your work with a Defensics testplan. The best way to understand this is to try it.
Stop your test run first by clicking on the stop button in 6) Test run
or the stop button in the
toolbar.
Choose File > Save testplan. Enter a name, then click Save.
Now quit Defensics and start it again. Choose File > Open testplan and select the testplan you just saved.
Defensics loads the same test suite (or suites) you were using and configures everything in exactly the same way.
Defensics testplans are extremely useful for repeatable testing. In addition, running a testplan automatically is easy.
In certain situations, it is also useful to save just the settings for a test suite. You can do this with File > Save settings. Similarly, you can load a set of settings with File > Load settings. Most of the time, a testplan does what you want, but settings files are occasionally useful.
3.8. About Benchmark
If you click on the triangle next to Benchmark
, it expands and shows another set of statistics.
This feature helps you compare the fuzzing that you’re doing with the rest of the Defensics community. You can see the number of test cases you’ve run, the time you’ve spent testing, and other statistics about your current test run, compared with the same statistics averaged from other Defensics users worldwide. Each time you finish a test run, Defensics carefully anonymizes these statistics about your test run and sends them to the Defensics cloud service. There, the statistics are decoupled from the originating
IP address before being stored. At no time does any information about your target or your found vulnerabilities leave your test machine.
16