and best practices Sunny SuenManaging Principal, AsiaESP Solutions ConsultingFlexConnector deep dive and best practicesAgendaFlexConnector deep dive•Customized event feeding options•Advanced topics in FlexConnector development FlexConnector best practices•Essential steps on FlexConnector configuration & development•Best Practice of FlexConnector submission to improve –Maintainability–Readability–Efficiency–Accuracy Target audienceGo through FlexConnector training and documentationHave created FlexConnector in practical environment ReferenceDocumentation•FlexConnector Development Guide•ArcSight Categorization Technical Note•Other SmartConnector Configuration GuideTraining•FlexConnector TrainingIntroductionSmartConnector architectureIDSFirewallUnix SyslogArcSight managerSmartConnector•An application that collects raw events from security devices, processes them into HP ArcSight security events, and transports them to destination devices. •SmartConnectors are generally one of the following types –File Connectors–Database Connectors–API Connectors–SNMP Connectors–Microsoft Windows Event Log Connectors–SyslogConnectors–ScannerConnectors–FlexConnectors–ModelConnectors FlexConnector•The FlexConnector framework is a software development kit (SDK) that lets you create a SmartConnector tailored to the devices on your network and their specific event data. •The available FlexConnectors are:–Logfile FlexConnector (fixed-format)–Regex FlexConnector (variable-format)–Database FlexConnectors–SNMP FlexConnector–Syslog FlexConnector–XML FlexConnector–Scanner FlexConnector–REST FlexConnector–Key-value FlexConnector (via Logfile/RegexFlexConnector)SmartConnector vs. FlexConnectorDo we need FlexConnector?First, capture the requirement•Capture all device log details–Vendor Name (e.g. ABCDE Technologies)–Product Name (e.g. ABCDE Web Server)–Software/Firmware version (e.g. version X.X)–Log Type (e.g. flat file)–Log format / transport (Free format text log)–Log rotation scheme (Daily; filename containing event log date: AccessLog_yyyyMMdd.log) –Event Type (e.g. access audit log)•Match supported device list of SmartConnector–If not in the list, seek for following options…Do we need FlexConnector?Options for customized event feeding•FlexConnector Development–For a complete development of parser/categorization on all required events –Identify FlexConnector format / transport•Reuse of SmartConnector parser/categorization–Partial development for unparsed/uncategorized events–Identify the similar type of parser/categorization•Map files/External Mapper–For further interpretation some event values–E.g. elaborating department name “HR” to “Human Resource Department” •Common Event Format (CEF)–Add/modify application log format as CEF outputFlexConnector developmentType of FlexConnectors•Logfile FlexConnector (fixed-format)•Regex FlexConnector (variable-format)•Database FlexConnectors•SNMP FlexConnector•Syslog FlexConnector•XML FlexConnector•Scanner FlexConnector•REST FlexConnector•Key-value FlexConnector (via Logfile/Regex FlexConnector)Reuse of SmartConnector parserWhen I need this?To reuse (convert) standard SmartConnector’s parser of the required log transport type, such as •Convert a file-based FlexConnector to folder-based FlexConnector• A file-based FlexConnector is wrapped by syslog transportHow to use it?•Identify the folder/file name of standard parser file (from aup)unzip -l {$Connector}/current/system/agent/arcsightagents.aup– e.g. apache/apache_access_file•Scenario 1: Converting standard file reader Connector to multi-folder Connector–Configure Multi-Folder FlexConnector to assign the configfile as the standard parser path agents[0].foldertable[0].configfile=apache/apache_access_file •Scenario 2: Converting standard file reader Connector to syslog transport–Create Syslog FlexConnector with following extraprocessor statementextraprocessor[0].type=regexextraprocessor[0].filename=apache/apache_access_fileReuse of SmartConnector categorizationWhen I need this?To reuse the standard categorization file of one supported device to the FlexConnector How to use it?•Check in the agent.log on the categorization file being used•Identify the folder/file name of standard categorization file (from aup)unzip -l {$Connector}/current/system/agent/arcsightagents_{date-version}.aup –e.g. apache/apache.csv•Create additional categorization file, e.g. newvendor/newproduct.csv•Include the folder/file name of standard categorization file into the link file, e.g.–Create a file newvendor/newproduct.link.csv in the acp/categorizer/current folder–Add following lines into newproduct.link.csv•/apache/apache.csv•/newvendor/newproduct.csvMap file revisitWhen I need this?To map additional values from event field(s) to assign new values How to use it?•create map.n.properties (where is 1,2,3,etc in sequence)•put getter and setter mapping on the first line of the map.n.properties file, such as –Line 1: Getters and setters•event.deviceHostName,set.event.deviceCustomString1–Line 2 onwards: value mapping•Host1,HR Dept•Host2,FIN DeptType of getters•Exact match: Header Row=event.deviceHostName, data=Host1•Range: Header Row=range.event.destinationPort, data=10000-19999 •Regex: Header Row=regex.event.deviceHostName, data=HR.*When I need this?Perform external database query on event field for additional mapping information How to use it?•Check the agent URI (with 2 ‘equal sign (==)’ suffices)•Create the folder: user/agent/extmap/{agent URI}•Create the external mapper file extmap.n.properties (where n=1,2,3… in sequence) into this folder(Continued)•Add following lines into the extmap.n.propertiestype=sqlfield.getter=deviceAddressfield.setter.count=1field.setter[0]=deviceCustomString1#field.addrs.as.numbers=truejdbc.class=org.gjt.mm.mysql.Driverjdbc.url=jdbc:mysql://localhost:3306/threatIntelername=inteljdbc.password=OBFUSCATE.4.8.1:LOX6GXaJ+5imr6M1wmwkNg==jdbc.query=select address, threatVector from watchList WHERE address in (?\u0000?) •The password has can be created by this command./arcsight agent runjava com.arcsight.agent.loadable._ExternalMapperComponent -p 'arcsight'Type of FlexConnectorsCEF formatWhen I need this?To adapt the application log to ArcSight CEF format. This can eliminate the FlexConnector parser maintenance afterwardsHow to use it?•Obtain the “Common Event Format” Documentation•Adapt the application log output to CEF format•Create event categorization file based for all possible event types HP ArcSight Common Event FormatAdvanced topics in FlexConnector development Scope•Advanced techniques commonly encountered in practical environment•Not currently discussed in documentation in detailsTopics•Fragmented event lines•Character encodingFragmented event linesWhen I need this?Some devices send single event in multiple log linesTo merge the information of all the related events into a single one Handling options•multiline.regex•Regex (?s)•Line.ignore.regex•Event mergingWhen I need this?In some instances the events sent by the device will not necessarily be close together, there could be events that will be sent in between other events, such asHow to use it?Use merge parameters:merge.count=1merge[0].pattern.count=2merge[0].pattern[0].token=NAME1merge[0].pattern[0].regex=(BIND|UNBIND|MOD|R ESULT)merge[0].pattern[1].token=NAME2merge[0].pattern[1].regex=(BIND|UNBIND|MOD|R ESULT)2merge[0].starts.count=1merge[0].starts[0].token=NAME3merge[0].starts[0].regex=(BIND|UNBIND|MOD) merge[0].ends.count=3merge[0].ends[0].token=NAME4 merge[0].ends[0].regex=RESULT merge[0].ends[1].token=NAME5 merge[0].ends[1].regex=RESULT2 merge[0].ends[2].token=NAME6 merge[0].ends[2].regex=RESULT3 merge[0].timeout=60000 merge[0].id.tokens=conn|msgId merge[0].id.tokens.delimiter=| merge[0].sendpartialevents=true merge[0].capacity=100(Continued)Character encodingWhen I need this?The raw log data contains non-ASCII charactersHow to use it?Configuration: agent.properties•The Multi Folder Follower FlexConnector–agents[0].foldertable[0].encoding=UTF-16LE•SNMP FlexConnector–snmp.charset={Your character set for the foreign language} (such as snmp.charset=big5) Configuration: JVM option•For other FlexConnector (such as Syslog FlexConnector)–In ${CONNECTOR}/current/bin/scripts/connectors.sh (or connectors.bat), append following options in the “ARCSIGHT_JVM_OPTIONS”:-Dfile.encoding=(character encoding) (such as -Dfile.encoding=gb2312)Character encoding(Continued)Development: regex–main / submessage regex statement•convert non-ascii/multi-byte characters into unicode notation, such as \u7528\u6236 (Chinese characters of ‘User’)••Folder structure FlexConnector Configuration ${ArcSight Connector Home}/current/user/agent User Development Folders •flexagent: parser•acp: event categorization•fcp: standard parser / parser override•map: map file•extmap: external mapper•lib: jdbc driverConfigurationagent.propertiesLocation under {$Connector_Home}/current•user/agentWhen I need it?To tune the advanced configuration parameters, such as:•File Rotation•Filename extractor•Subagent listRefer to agent.default.properties for more optionsHow to use it?•For Software Connector, modify the file directly in user/agent/agent.properties •For Connector Appliance, use “Diagnostic Wizard” and select agent.propertiesparser (*.properties)Location under {$Connector_Home}/current •user/agent/flexagentSections in properties file•Parser configuration•Token Declaration•ArcSight event field assignment•Severity mapping•Conditional mapping•ExtraprocessorPoint to Note•Proper comment should be required–Sample messages–Message group–Section headingFile extension and Locationflexagent/•Log-file: <product_name>.sdkfilereader.properties•Regex Log-file: <product_name>.sdkrfilereader.properties•Regex Folder Log-file: <product_name>.sdkrfilereader.properties•XML Folder Log-file: <product_name>.xqueryparser.properties•Time-based Database: <product_name>/<table_name>.sdktbdatabase.properties•ID-based Database: <product_name>/<table_name>.sdkibdatabase.properties •Multi-Database: <product_name>/table <n>/<table_name>.sdktbdatabase.properties •Syslog: s yslog/<product_name>.subagent.sdkrfilereader.properties •SNMP: <vendor>/sdksnmp.#.snmptrap.properties (where # = trap type) •Scanner (Normal Text): <vendor>.scanner.sdkrfilereader.properties•Scanner (XML): <vendor>.scanner. xqueryparser.properties•Scanner (Database): <vendor>. sdkdatabase.properties•REST: <product_name>.jsonparser.propertiesCategorization (*.csv)Location under {$Connector_Home}/current•user/agent/acp/categorizer/current/<device_vendor>/<device_product>.csvFile formatGetters and setters•Exact match, range, regex getters are all supportedSingle categorization file: vendor1/product1.csvMultiple categorization file: vendor1/product1.link.csv, vendor1/product.csv and vendor1/product.0.csv Point to NoteFile name capitalization•All folder and file names are in small letters•Space or non-alphanumeric letters are converted into underscore ‘-’, such as–AS/400 -> as_400–Microsoft Windows -> microsoft_windowsCategorization (*.csv) (Continued)Point to Note (continued)•Try using Regex getter and put some catch all to enhance the categorization result, such as–Logon.* (For all undefined event type with “Logon” prefix)–.* (For all other undefined event type)•Required categorization fieldsregex.event.deviceEventClassId,set.event.categoryObject,set.event.category Behavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.eve nt.categoryDeviceType,set.event.categoryOutcome,set.event.categorySignific ance,set.event.agentSeverity•Obtain the ArcSight Categorization Technical NoteKey field assignmentTopicsTime fields:•deviceReceiptTime, startTime, endTimeDevice name for categorization link:•deviceVendor / deviceProductEvent type:•name/deviceSeverity/deviceAction/deviceEventClassIdCustom fields•Use deviceCustom* and avoid flex*•flex* fields are reserved for console mapping to additional dataSource-destination vs attacker-target•Only use source/destination pair (e.g. sourceUserName/destinationUserName) •attacker/target pair: result of correlation, not to be assigned in FlexConnector Assignable fields (list of assignable fields in FlexConnector Developer’s Guide)Local testing (CSV/CEF output)When I need this?To test and debug the parser/categorization in the development phase, it will be more efficient to verify locally rather than submitting to logger and ESMHow to use it?•For connector destination, choose CEF log file or CSV log file•Select header row to help readablility•For CSV log file, you can choose the required fields as outputPackagingFor submission of FlexConnector for production usage, we recommend: •Single Package file•Folder structure following Connector installation standard•Put device vendor/device product/FlexConnector version number in the package name ExamplePackage: $deviceProduct_flexconnector_vx.x_yyyy-mm-dd.zipContent folder structure:acp/fcp/flexagent/DocumentationHighly recommend to provide following documentation in the FlexConnector files for ongoing maintenance:•Parser comments•Configuration DocumentationPlease fill out a survey.Hand it to the door monitor on your way out. Thank you for providing your feedback, which helps us enhance content for future events.Session TB3033 Speaker Sunny Suen Please give me your feedback。
