DESIGN, DEVELOPMENT, AND APPLICATION OF LIDAR DATA PROCESSING TOOLS

In presenting this thesis in partial fulfillment of the requirements for an advanced degree at Idaho State University, I agree that the Library shall make it freely available for inspection. I further state that permission for extensive copying of my thesis for scholarly purposes may be granted by the Dean of the Graduate School, Dean of my academic division, or by the University Librarian. It is understood that any copying or publication of this thesis for financial gain shall not be allowed without my written permission.Signature ___________________________________Date _______________________________________DESIGN, DEVELOPMENT, AND APPLICATION OF LIDAR DATA PROCESSING TOOLSbySara E. EhingerA thesissubmitted in partial fulfillmentof the requirements for the degree ofMaster of Science in Geographic Information ScienceIdaho State UniversityJune 2010To the Graduate Faculty:The members of the committee appointed to examine the thesis of Sara E. Ehinger find it satisfactory and recommend that it be accepted._________________________________Dr. Nancy F. GlennMajor Advisor_________________________________Dr. Daniel P. AmesCommittee Member_________________________________Dr. Temuulen T. SankeyCommittee Member_________________________________Dr. Judith A. CrewsGraduate Faculty RepresentativeAcknowledgementsI thank my major advisor, Dr. Nancy Glenn, for her endless great ideas and for making each goal seem so manageable. Thanks to Dr. Teki Sankey for getting me started, sentence by sentence, and for a lot of support along the way. And thanks to Dr. Dan Ames for his programming enthusiasm and advice. Dr. Judith Crews served as the graduate faculty representative and was a positive addition to the committee. Lucas Spaete provided writing, editing, and data analysis help; and Rupesh Shrestha shared IDL knowledge. My supervisors at the Boise National Forest, Mike Williamson and Carey Crist, helped me balance work and school. Patrick Kormos kept me going in the last few weeks with creative threats of fasting and kittens. Finally, I would like to thank my parents, brother, sister-in-law, family, and friends for their support. My research and education were funded by the NSF Idaho EPSCoR Program and the National Science Foundation under award number EPS-0814387, NOAA Earth System Research Laboratory Physical Sciences Division Grant #NA06OAR4600124, the BLM Owyhee Uplands Pilot Project (ISU-BLM Agreement #DLA060249), and the AmeriCorps Education Award.Table of ContentsPhotocopy and Use Authorization (i)Title Page (ii)Committee Approval (iii)Acknowledgements (iv)Table of Contents (v)List of Figures (vii)List of Tables (viii)Thesis Abstract – Idaho State University (2010) (ix)CHAPTER 1: INTRODUCTION (1)1.1 LiDAR Overview (1)1.2 Statement of Purpose (2)1.3 BCAL LiDAR (3)1.4 LAS Format (4)1.5 IDL Programming Overview (5)1.6 Reynolds Creek Experimental Watershed LiDAR Data (8)CHAPTER 2: LITERATURE REVIEW (10)2.1 LiDAR Applications (10)2.2 Other LiDAR Software Solutions (11)2.3 BCAL LiDAR Applications (13)2.4 LiDAR Processing and Analysis with IDL (14)CHAPTER 3: LAS FORMAT TO ASCII FORMAT TOOL (15)3.1 Introduction (15)3.1.1 Tool Overview (15)3.1.2 Uses of ASCII LiDAR Data (15)3.1.3 Existing Software for Converting Between LiDAR File Formats (18)3.1.4 Problem Statement (19)3.2 Methods (20)3.2.1 Design (20)3.2.2 Graphical User Interface (21)3.2.3 User Interactions (22)3.2.4 Selected IDL Code (23)3.3 Results (25)3.4 Discussion (27)CHAPTER 4: VEGETATION HEIGHT GROUPS TOOL (30)4.1 Introduction (30)4.1.1 Tool Overview (30)4.1.2 Vegetation Height Groups Literature and Applications (30)4.1.3 Problem Statement (31)4.2 Methods (32)4.2.1 Design (32)4.2.2 Graphical User Interface (33)4.2.3 User Interactions (34)4.2.4 Selected IDL Code (35)4.3 Results (38)4.4 Discussion (43)CHAPTER 5: INTERPOLATE GROUND POINTS TOOL (44)5.1 Introduction (44)5.1.1. Tool Overview (44)5.1.2 LiDAR Interpolation Literature Review (45)5.1.3. Problem Statement (47)5.2 Methods (48)5.2.1 Design (48)5.2.2 User Interface (48)5.2.3 User Interactions (50)5.2.4 Selected IDL Code (51)5.3 Results (53)5.4 Discussion (57)CHAPTER 6: CONCLUSIONS (60)REFERENCES (62)APPENDICES (67)Appendix A: LAS to ASCII User Instructions (67)Appendix B: Vegetation Height Groups User Instructions (68)Appendix C: Interpolate Ground Points User Instructions (69)Appendix D: Update to Height Filtering Tool (70)Appendix E: Export Ground Points Only (LAS) tool (72)Appendix F: Common IDL Functions and Procedures in BCAL LiDAR (73)Appendix G: Toolkit and Source Code Download (77)Figure 1. BCAL LiDAR added in ENVI as an extension. (3)Figure 2. ReadLAS_BCAL.pro hierarchy chart. (7)Figure 3. LiDAR-based classification of juniper presence and absence (Sankey et al., in press). (17)Figure 4. LAS to ASCII program hierarchy chart. (21)Figure 5. a) LAS to ASCII user options GUI and b) Choose output folder dialog. (22)Figure 6. LAS to ASCII user interaction process. (23)Figure 7. LAS to ASCII conversion report. (25)Figure 8. ASCII format LiDAR point data records a) opened in Notepad and b) imported to Excel. (26)Figure 9. Vegetation Height Groups program hierarchy chart. (33)Figure 10. Vegetation Height Groups GUI. (34)Figure 11. Vegetation Height Groups user interaction process. (35)Figure 12. Vegetation Height Groups processing report. (39)Figure 13. Results of vegetation height classification as seen in the user interface. (40)Figure 14. Vegetation Height Groups displayed in the BCAL LiDAR 3D Visualizer (41)Figure 15. Point density rasters with aspen stands in yellow and conifer stands in green. The red circle indicates an area of high point density due to flight line overlap. (42)Figure 16. Interpolate Ground Points program hierarchy chart. (48)Figure 17. Interpolate Ground Points user options GUI. (50)Figure 18. Interpolate Ground Points user interaction process. (51)Figure 19. Interpolate Ground Points processing report. (54)Figure 20. Hillshade images (200 m x 200 m) of four rasterization methods: a) Rasterize LiDAR tool (nearest neighbor-type); and those developed in the tool described herein: b) natural neighbor; c) linear; and d) quintic (55)Table 1. Original BCAL LiDAR tools and their descriptions. (3)Table 2. LAS version 1.x point data record items, formats 0 and 1. (8)Table 3. LiDAR Software. (12)Table 4. Procedures used in the LAS to ASCII tool. (24)Table 5. LAS to ASCII conversion results for seven test LiDAR datasets using BCAL LiDAR, FUSION, and LAStools. (29)Table 6. Procedures and functions used in the Vegetation Height Groups tool. (36)Table 7. Interpolation options in the BCAL height filtering tool. (47)Table 8. Procedures and functions used in the Interpolate Ground Points tool. (51)Table 9. Accuracy evaluation using the Mean Signed Error (MSE) and Root Mean Squared Error (RMSE) in meters of four bare earth interpolation methods as compared to field measured bare earth elevation values (n = 636). (56)Design, Development, and Application of LiDAR Data Processing Tools Thesis Abstract – Idaho State University (2010)Processing Light Detection And Ranging (LiDAR) data remains a significant challenge in many applications, including vegetation characterization and hydrologic modeling. The demand for free and open source LiDAR processing software is increasing as more LiDAR datasets are available and the price of new data acquisitions decreases. BCAL LiDAR, a suite of LiDAR processing tools developed at the Boise Center Aerospace Lab of Idaho State University, helps fill this need. The software, written in the Interactive Data Language (IDL), has point data analysis capabilities including subsetting, filtering, classification, and rasterization. BCAL LiDAR was developed for small footprint, discrete return, airborne LiDAR with a focus on vegetation and terrain applications. This research expanded BCAL LiDAR with the addition of three new tools: LAS to ASCII Format, Vegetation Height Groups, and Interpolate Ground Points. The conversion of LiDAR data from the LAS binary format to human-readable text format allows examination of point attribute data and importing into commonly used software. The second tool provides the ability to segregate and analyze LiDAR point data within specified vegetation height groups. Finally, the Interpolate Ground Points tool adds new interpolation methods for creating bare earth elevation rasters since no one method is best for all applications. All new tools were tested with and applied to a LiDAR dataset collected in November 2007 over the Owyhee Mountains of southwestern Idaho, U.S.A. BCAL LiDAR, including the three new tools, is available for download from the BCAL website () at no cost with source code included.CHAPTER 1: INTRODUCTION1.1 LiDAR OverviewLight Detection And Ranging (LiDAR) is an active remote sensing technology. Sensors can be airborne, space-borne or ground-based. In the airborne LiDAR system, a laser pulse is emitted from the plane-mounted sensor to the earth‟s surface. The reflection of that pulse from the ground or vegetation is recorded at the sensor along with the round-trip travel time (Lefsky et al., 2002). The time interval multiplied by the speed of light is used to calculate the distance between the sensor and the target. Many laser scanners also record multiple returns when a single pulse is scattered by targets in its path (Wehr & Lohr, 1999). The global positioning system (GPS) and inertial measurement unit (IMU) data recorded on the plane are combined with the laser pulse range measurements to produce point data with x, y-locations and elevation (ASPRS, 2005). Helicopters, air balloons, and unmanned aerial vehicles can also carry airborne LiDAR sensors. Alternative to discrete lasers, full-waveform sensors record the full energy return (echo) from the ground surface and are used to capture complete elevation profiles of objects on the ground surface.LiDAR data have been applied in geomorphology, silviculture, and forest ecosystem sciences (Vierling et al., 2008). The use of LiDAR continues to increase as LiDAR datasets are made available to the public and the price of new data acquisitions decreases (Chen, 2007). Despite the varied applications and availability of data, LiDAR processing remains a significant challenge (Chen, 2007). Height filtering, the process of classifying the data into ground and non-ground returns, is an essential but difficult step in most analyses (Liu, 2008).LiDAR provides detailed topographic data over broad areas. The elevation data can be used in the creation of DEMs (Digital Elevation Models), DSMs (Digital Surface Models), DTMs (Digital Terrain Models), or in applications such as vegetation characterization and hydrological modeling. However, preprocessing of the raw LiDAR data is required before such analyses can be performed.1.2 Statement of PurposeA suite of LiDAR processing tools has been developed at the Boise Center Aerospace Laboratory (BCAL) of Idaho State University. The tools, referred to as BCAL LiDAR, automate common processing tasks, such as file format conversion, subsetting, height filtering, and rasterization, with a focus on vegetation and terrain applications. The purpose of this thesis is to expand the analysis and research capabilities of the toolkit. This research resulted in the creation of three new LiDAR tools. Chapter 3 describes the LASer (LAS) file format to ASCII conversion tool, Chapter 4 describes the Vegetation Height Groups tool, and Chapter 5 describes the Interpolate Ground Points tool. For each of these chapters, the introduction provides an overview of the tool, applications, relevant literature, and a specific problem statement. The methods sections include the following steps of the program development cycle: design, user interface, and selected Interactive Data Language (IDL) code (Schneider, 2005). Evaluation and testing are described in the results and discussion sections of each of the three chapters. User instructions are included in the appendices. Within the wide variety of available LiDAR processing software, these tools are unique because they are free, open source, extensible, and run from a simple user interface.1.3 BCAL LiDARBCAL LiDAR is a free extension for ENVI image analysis software. ENVI can be extended or modified with Interactive Data Language (IDL version 7.1, ITT Visual Information Solutions, 2009). Both ENVI and IDL are products of ITT Visual Information Solutions (ITT Visual Information Solutions, Boulder CO). BCAL LiDAR is available for free download from the BCAL website () and the ENVI extensions site (). When installed, the BCAL LiDAR menu is added to the main ENVI toolbar (Figure 1).Figure 1. BCAL LiDAR added in ENVI as an extension.The original BCAL LiDAR menu, published in 2005-2006, contained 14 functions. A list of these tools and descriptions are shown in Table 1.Table 1. Original BCAL LiDAR tools and their descriptions.Tool DescriptionGet LAS File Info Displays header and projection information for an LASfileConvert ASCII Data to LAS Converts text point data into LAS format LiDAR files Rasterize LiDAR Data Converts raw point data into various raster products Add Projection to LASAdds embedded projection information to LAS file Files(s)Reproject LAS File(s) Converts existing LAS files into a new map projection Create Boundary EVF Creates an ENVI Vector File that shows the boundariesof the dataPerform Height Filtering Performs vegetation filtering on the dataCreate Elevation Profiles(s) Creates one or more elevation profiles along a user-defined transectBuffer LAS Files Geographically buffers the data using neighboring files Decimate LAS File(s) Reduces the number of points in an LAS file byrandom decimationSubset via Coordinates Subsets data files according to user coordinates Subset via Image/ROI Subsets data files using a reference image and/ordifferent regions of interestTile LAS Files(s) Divides one or more files into multiple tiles3d LiDAR Viewer An interactive viewer for displaying LiDAR point dataBCAL LiDAR is currently used in research at Idaho State University, Boise State University, the USDA Agricultural Research Service in Boise, and is anticipated to be used by other agencies. ENVI+IDL users can implement the tools, and view or edit their source code. The code is open source, meaning that the BCAL LiDAR binaries and source code are freely available under an open source license as per the Open Source Initiative (), and can be downloaded and used for commercial and noncommercial activities. However, the code uniquely runs on ENVI and IDL. Open access to LiDAR analysis tools encourages use by a broad range of scientists (Vierling et al., 2008). Sharing the BCAL LiDAR tools allows others to make improvements and expand their capabilities for specific user needs.1.4 LAS FormatThe LASer (LAS) file format and file extension are unique to LiDAR data and follow a binary format developed by the American Society for Photogrammetry and Remote Sensing (ASPRS) LiDAR Committee. LAS provides a common format for LiDAR point data to allow sharing between different vendors, users, software, and hardware (ASPRS, 2005). The ASPRS LiDAR Committee manages the LAS file specifications (ASPRS, 2010). The current version, LAS 1.3 was approved by the ASPRS Board in 2009. Prior to the development of LAS version 1.0 in 2003, there was not a standardized format for LiDAR data. Data were commonly stored as ASCII text files with different products including or excluding now standardized information. The binary LAS format provides efficient storage of large datasets and allows for faster processing than text file format.The LAS format consists of a public header block, one or more variable length records, and point data records. Version 1.3 defines five different point data record formats (ASPRS, 2009). Formats 0 and 1 have remained similar through each version of the LAS specifications. Point data record formats 2 and 3, added in version 1.2, provide support for ancillary image data with the addition of Red, Green, and Blue fields for spectral data in the respective wavelengths. LAS version 1.3 added formats 4 and 5, along with an extended variable length record, to store full-waveform LiDAR data. BCAL LiDAR currently supports point formats 0 and 1 of LAS versions 1.0, 1.1, 1.2, and 1.3.1.5 IDL Programming OverviewIDL is a high-level programming language specializing in the interactive visualization and analysis of large datasets that draws some features from Fortran and C (Bowman, 2006). Functions and procedures in IDL are saved as .PRO files. These programs can also be compiled as .SAV files. All tools in the BCAL LiDAR suite are distributed in both formats. SAV files are executable in ENVI without an IDL license, but this format hides the source code. By providing both .PRO and .SAV formats, the tools are available to a wider range of users as many ENVI users do not have access to the IDL license. The .PRO distribution of the files contains the source code which can be read and edited in IDL or any text editor.In IDL, graphical user interfaces (GUIs) are created with a set of objects called widgets. ENVI includes a series of widgets and an auto-manage function that allow for relatively simple creation of GUIs (see Appendix F). However, the ENVI auto-managed widgets provide limited user interaction options. The IDL commands, while significantlymore complicated to implement, allow for more varied graphical layouts with sophisticated user interaction. GUI use depends on the tool. For the LAS to ASCII and Interpolate Ground Points tools, which only require a few user input parameters before the procedure is executed, the auto-managed ENVI widgets are sufficient. However, the Vegetation Height Groups tool has an interactive user interface written with IDL widgets.The first step in all three of the tools is to load the selected LAS input file. This task is accomplished by reading the three parts of the binary LAS file, namely the public header block, the variable length records, and the point data records to IDL structure variables. The point data records are stored in an array of structures, one for each LiDAR point return. Figure 2 shows the hierarchy chart for the ReadLAS_BCAL procedure that is called by each of the tools.Figure 2. ReadLAS_BCAL.pro hierarchy chart.The point data structure created by the ReadLAS_BCAL procedure contains either 12 or 13 fields, depending on the format of the input LAS file. Table 2 provides the LAS field names, the header names exported by the LAS to ASCII tool, the IDL structure variable field names, and the IDL data types. The program for each tool follows the same general structure: 1) menu button definition, 2) set compile options, 3) establish an error handler, and 4) execute the main program.Table 2. LAS version 1.x point data record items, formats 0 and 1.LAS Point Data Record Item LAS to ASCIIHeader Row NameIDL Structure TypeX X_Easting data.east Integer (0L)Y Y_Northing data.north Integer (0L)Z Z_Elevation data.elev Integer (0L) Intensity Intensity data.inten Integer(0US)Return Number* ReturnNum data.nReturn mod 8 Byte (0B) Number of Returns(given pulse)*NumOfReturns floor(data.nReturn/8) mod 8 Byte (0B)Scan Direction Flag* ScanDirFlag floor(data.nReturn/64) mod 2 Byte (0B) Edge of Flight Line* EdgeFlightLine floor(data.nReturn/128) mod 2 Byte (0B) Classification Classification data.class Byte (0B) Scan Angle Rank ScanAngleRank data.angle – (256 *(floor(data.nReturn/128) mod 2))Byte (0B)User Data UserData er Byte (0B) Point Source ID PointSourceID data.source Integer(0US)GPS Time** GPS_Time data.time Double (0D)* These four fields (return number, number of returns, scan direction flag, and edge of flightline) are all stored within a single byte value.** The GPS Time is only included if the LAS file uses point data record format 1.1.6 Reynolds Creek Experimental Watershed LiDAR DataThe tools developed in this research were applied to and tested with a LiDAR dataset that was collected as part of a regional acquisition jointly funded by Idaho State University, University of Idaho, USDA Natural Resources Conservation Service, and the Bureau of Land Management. The data were collected by Watershed Sciences (Corvallis, OR) over 430 km2 in 5 southwest Idaho study sites between November 10-18, 2007 (Watershed Sciences, 2008a,b). The vendor used a Leica ALS50 Phase II laser which can record up to four returns from a single pulse. Unclassified LiDAR points were delivered in LAS v1.1 format. The data from the Reynolds Creek study site (240 km2) were used exclusively in this thesis. As reported by the vendor, the average point density in this dataset was 5.65 points/m2 with a vertical accuracy 3.4 cm. Mean relative accuracy was 0.03 m and absolute accuracy was 0.03 m (1 σ). The vendor collected 1002real-time kinematic (RTK) ground survey points that were used for the accuracy assessment. A comparison of x, y, and z locations with a RTK to a subset of the Reynolds Creek LiDAR dataset indicated that vertical and horizontal accuracy are approximately 10 cm and 30 cm, respectively (Glenn et al., in review).Field data in the Reynolds Creek Experimental Watershed (RCEW) were collected by the USDA Agricultural Research Service Northwest Watershed Research Center in 2007 and 2008. BCAL researchers assisted with data collection during summer and fall of 2008. The site is located approximately 80 km southwest of Boise in the Owyhee Mountains. The RCEW is a long term hydrologic study area established in 1960 (Slaughter et al., 2001). Sagebrush and grassland are the dominant plant communities with Douglas fir, subalpine fir, and aspen in areas with more snow accumulation (Slaughter et al., 2001). Current projects using the RCEW dataset include biomass calculations and LAI measurements. Also the data were used to examine juniper encroachment (Sankey et al., in press), aspen decline in the area (Sankey, in review), shrub community type classification (Sankey and Bond, in review), sagebrush height and area estimation (Glenn et al., in review), and bare earth accuracy (Spaete et al., in review).CHAPTER 2: LITERATURE REVIEW2.1 LiDAR ApplicationsLiDAR data provide valuable information about the three-dimensional structure of vegetation. Forest structure parameters, including tree height (Andersen et al., 2006), forest basal area and tree density (Hudak et al., 2006), and leaf area index (Jensen et al., 2008) have been accurately measured with airborne LiDAR data. Vierling et al. (2008) reviewed recent habitat characterization studies using LiDAR. Two examples are the correlation of vegetation structure diversity to bird species diversity and canopy height as an indicator of bird habitat quality. Other LiDAR-derived metrics such as understory or middlestory height, roughness, volume, and stand density may be valuable in broad-scale landscape analysis for species management decisions and conservation planning.LiDAR data points grouped into evenly spaced grid cells for statistical analysis can be combined with co-registered reflectance data. Varga & Asner (2008) combined the maximum vegetation height from each pixel with the results of an automated mixture analysis from hyperspectral data to create a new fire fuels index. The resulting values provided more accurate fire hazard data than possible with either LiDAR or reflectance data alone. LiDAR data can also be combined with simultaneously collected orthophotos to improve the filtering of ground points, especially in areas of small shrub cover (Riaño et al., 2007).Another important use of LiDAR data is DEM generation for terrain modeling applications (Liu, 2008). The creation of bare earth and canopy surface models requires the challenging process of classifying LiDAR point data into ground and non-ground returns (Lefsky et al., 2002; Meng et al., 2010). Filtering algorithms includeinterpolation-based, slope-based (Popescu et al., 2002) and morphological filters (Chen et al., 2007; Zhang & Whitman, 2005). The multiscale curvature algorithm (MCC) employs an iterative thin plate spline interpolation with changing scale and curve tolerance parameters to identify non-ground returns (Evans & Hudak, 2007). The multi-directional ground filtering (MGF) algorithm is an example of a directional filtering method (Meng et al., 2009). After finding the lowest pixel in a local neighborhood to set the initial ground point, the MGF scans a 2-dimensional neighborhood and compares LiDAR point elevation differences to a set elevation threshold in this iterative ground point labeling process. The BCAL LiDAR height filtering algorithm uses an iterative ground point identification process with local interpolation (Streutker & Glenn, 2006). The LiDAR point data are divided into evenly spaced grid cells. An initial ground surface is interpolated using the lowest point in each of the grid cells. Any non-ground returns below the initial ground surface are then reclassified as ground, and a new surface is interpolated. This step is repeated until all points have been classified as ground or non-ground.2.2 Other LiDAR Software SolutionsThere are multiple other efforts focused on LiDAR data analysis that are in varied stages of development and application. While this is not an exhaustive list, the products include: FUSION, OPALS, Tiffs, and GeoLiTE. Table 3 shows a comparison of these four programs and BCAL LiDAR based on the following functionalities: 1) 3D point cloud visualization, 2) inclusion of a ground point filtering algorithm, 3) interpolation functions to create raster products from LiDAR point data, 4) feature extraction capabilities, 5) open source code, and 6) the platform for running the processing tools.Table 3. LiDAR Software.Software Visualize FilterPoints Inter-polateFeatureExtractionOpenSourcePlatformBCAL LiDAR Yes Yes Yes No Yes ENVI+IDL FUSION Yes Yes Yes Yes No command line OPALS No Yes Yes Yes No variousTiffs Yes Yes Yes Yes No standalone GeoLiTE Yes No Yes No No ArcMap FUSION, a free software package created by Robert McGaughey of the US Forest Service, provides LiDAR processing capabilities including bare-earth point filtering and surface fitting (USFS, 2010). The utilities are run from the command line and intended for forestry-related projects. The software also includes a standalone data visualizer with a graphical user interface.OPALS, Orientation and Processing of Airborne Laser Scanning data, is a product of the Vienna University of Technology (2010). The design uses a collection of small modular programs accessed from command line executables, Python modules, or via C++ API (application programming interface). The command line programs are recommended as the most straight-forward way to use the modules. The OPALS software aims to provide a complete LiDAR data processing chain, including point cloud classifications and DTM generation. A free version of the software, with a limit of 1 million LiDAR data points for processing, is available for use in small scientific studies.Chen (2007) created software called Tiffs (Toolbox for Lidar Data Filtering and Forest Studies) focused on filtering point cloud data and extracting individual tree structure information. Available tree parameters include height, crown area, and biomass. After the data is filtered the software can generate bare earth surfaces. Also。