EDUCATION高级数据共享技术：数据共享 技术的定义、原理及应用，以 技术的定义 原理及应用 以 及基于数据块的实现演讲人:刘松涛 ADIC高级技术顾问AbstractEDUCATIONAdvanced Data Sharing Technologies – Part 1 The What, Why and How of Data Sharing technologies plus block and filebased approaches for IT Director and Managers, IT/Storage/System Eng., Admins, Architects g g y g and Trainers How to deliver more performance and data accessibility with little to no additional cost? How to take advantage of existing storage infrastructure to provide more value to end-users and the global enterprise? A clear industry end users direction indicates that Data Sharing architectures and technology can be a good way to achieve these objectives. The first session offers a definition of Data Sharing and a discussion of its benefits with examples linked to the SNIA Shared Storage Model We cover the Model. main data sharing approaches and describe how they can improve performance, data accessibility, and manageability. This includes Scale-in and Scale-out methods based on block, file and application technologies such as: Cluster Volume M Cl t V l Managers, SAN File Systems, Cluster File Systems, Parallel Fil S t Cl t Fil S t P ll l NFS (pNFS), Object-based Storage Devices (OSD) and Global/Parallel File System.2SNIA Legal NoticeEDUCATION Th material contained i thi t t i l i copyrighted b The t i l t i d in this tutorial is i ht d by the SNIA Member companies and individuals may use this material in presentations and literature under the following conditions:– Any slide or slides used must be reproduced without modification – The SNIA must be acknowledged as source of any material used in the body of any document containing material from these y y g presentations This presentation is a project of the SNIA Education Committee3AgendaEDUCATION What is Data Sharing ?– Definition Why Data Sharing ?– End User Benefits How is Data shared ?– Block and file-based approaches file based4EDUCATIONWhat is Data Sharing ? Why Data Sharing ?What is Data Sharing ? gDefinition Shared access to same data (value & location) by multiple systems– Read/write: changes to data become visible to all servers – Read-only access via mechanisms that support shared read/write accessEDUCATIONExamples– Read/write access to a s a ed file is data s a g ead/ te shared e s sharing So is read-only access to a shared file– Clone/Snapshots (read-only or read/write) are not data sharing Changes do not affect original data Caching is data sharing when changes propagate– Changes to cached data must become visible to all – Other data changes must become visible via cacheReplication/CDP* is not data sharing because location changes– Potential divergence of data* CDP: Continuous Data Protection6Why Data Sharing ? y gEnd User Benefits Better performance and scalability– Larger server can be expen$$iveEDUCATION Sh i Sharing: apply more servers t same problem l to bl– Scales well for some applications, poorly for others – Can avoid replication or cloning Concurrency and Content access distribution– Use same data for more than one application Administration– Consolidated shared resource has lower TCO – Data Sharing increases the benefits of Storage/Data Consolidation7Storage Consolidation gScale-up by Scale-InEDUCATIONFile ServerNFS/CIFS ServerData Network - LANScale-InStorage NetworkShared Disks8Performance Improvement pScale-up by Scale-OutEDUCATIONApplication ServerApplication DB Engine Cluster Software Shared Storage Software…Scale-outStorage NetworkShared Disks9Why Data Sharing ? y gHow to apply Data Sharing to do useful things Hi h A il bilit Cl t High Availability Clusters (l (local & geographic) l hi ) Scaling applications– Web Servers - Read mostly/load balanced – Databases/OLTP/DW - Mostly use direct I/OEDUCATION Parallel applications and fast failover pp Systems and Applications Consolidation/Migration Off-host processing– Based on shared file system – Can also use by Point-in-Time copy techniques (not related to ou our data s a g de t o ) sharing definition) …address both Performance and Availability with no administration degradation and overhead10Some technologies and products …IBM AFSEDUCATIONWebNFSCisco FileEngineApple XsanVERITAS CFSDistributed, Samba FineGround Oracle OPS/RAC PolyServe Matrix Server ClusterLUSTRE or WebFS EMC HighRoad IBM Storage Tank IBRIX FusionFS SAN File System SGI CXFS CIFS DB2WAFS ONStor STOR-FS STOR-ISO9660CodaDiskSites FilePortRFSHP TruCluster/CFSParallel SUN SAM-FS &Sybase MPP Partitioned IBM SANergy Sanbolic Melio FS ApplicationsSMBADIC StorNext FSRedhat GFSNFSPPFSOpenAFSTacit Networks IsharedOSDPVFSSun QFS Volume WebNFSInformix XPSDFSpNFS11Isilon IQ OneFSNuview StorageXEDUCATIONHow i D t Sh d ? H is Data SharedHow is data shared ?Approaches and methods Several levels are possible– Share at the volume level (block based) – Share at the file or file system level (file block or object* based) (file, – Share at the database or application level (custom)EDUCATION In all cases, all these methods could occur– among like or dissimilar systems (OS), – concurrently or serially, – directly at the storage or in the network* For OSD13How is data shared ?Approaches and methods Traditional/Historical– Block level: Volume Management – File/File System (FS) level: Local FS (serial data sharing) and distributed methods with NAS, Samba, AFP (AppleShare), DFS, AFS/OpenAFS, RFS, Coda… – App./DB level: custom built-in methods (RDBMS, Email systems…) Check out SNIA T Tutorial: i lEDUCATION Advanced/Recent - File/FS level– Distributed: WAFS approach (NAS extension) and Network File Management/Virtualization (NFM), Global FS, SANFS and Cluster FS Storage VirtualizationCheck out SNIA Tutorial: NAS & iSCSI14How is data shared ?The SNIA Shared Storage Model Application li ti A li ti ApplicatioGFS n levelEDUCATIONFile/record layerDatabase (dbms)Shared LVMHost NetworkWAFS NASS rage Stor e dom n mainCluster FSFile system (FS)SAN FSBlock aggregationDeviceStorage devices (disks, …)Block l Bl k layer15Discovery, monitoring Resou urce mgm configuration mt, Security billing y, Redund dancy mg (back gmt kup, …) High a availabilit (fail-ov …) ty ver, C Capacity planning y gServ vicesHow is data shared ?Volume levelLVM LVMEDUCATIONData PathData Layout and OrganizationExamples– – – – – – – EMC PowerPath Volume Manager (PPVM) HP Shared Logical Volume Manager (SLVM) IBM Logical Volume Manager (LVM) MACROIMPACT SANique Cluster Volume Manager (CVM) REDHAT Logical Volume Manager (LVM) SANBOLIC LaScala VERITAS* [Cluster] Volume Manager (CVM/VxVM)* Merged with Symantec, July 200516How is data shared ?Volume levelEDUCATION Volume Managers allow data to be shared at a low level (block) without usually a built-in locking mechanism– Higher level applications control concurrent accesses to the data as needed – Can combine or divide physical resources (e.g., concatenation, mirroring, striping) and share the result – Volume Managers and the VTOC* problem VTOC Every OS has its own VTOC format Every VM has its own Volume Header and Table Definition – Same VM everywhere and y can share raw volume or same FS y you– Byte orders between processor Big Endian: Sparc, PA-Risc, Power – Little Endian: Intel)– Block s e o t e de ce a d b oc bou da y oc size on the device and block boundary could cause troubles – Concurrent or Serial access* Volume Table Of ContentsCheck out SNIA Tutorial:Storage Virtualization17How is data shared ?Volume level In-Band VirtualizationApplication Servers EDUCATION Out-of-Band VirtualizationApplication Servers Volume allocation ll tiIntelligent switch and/or ApplianceStorage Network Appliance Volume creationStorage NetworkShared DisksShared Disks18How is data shared ?Volume levelApplication Servers HPC App. HPC App. HPC App. …Shared Volume Manager - Storage SoftwareEDUCATIONHPC App.Storage S NetworkExample: pHPC* Application- How ? Shared DisksOwn data format on disk Own lock mechanism Increased throughput g p More effective use of servers* High Performance Computing- Benefits19How is data shared ?File/File System levelEDUCATION Share at File or File System (FS) level M lti l approaches & l Multiple h levels of maturity l f t it– Block-based Local (physical) Disk File System* for serial data sharing Disk based Cluster File System SAN File System or SAN File Sharing System– File-based or Distributed File System NFS/CIFS, WAFS and NFM** NFM Global, Parallel and Distributed FS* like UFS, HFS, XFS, JFS, VxFS, NTFS, ext2/3… ** Network File Management (also Network File Virtualization)20File/record layerEDUCATIONApplication A li Application level iF File/reco ord layerHos with LVM st. and software RAID s DHos with LVM st.Cluster FSHostHost LAN Dist. FS1. Direct-attach 2. SN-attach 3. NAS head 4 NAS server 4.SAN FS NAS headCluster FS NAS server Host block-aggregation Network block-aggregationShared LVM SNBlock la ayerDisk arraygg g Device block-aggregation21How is data shared ?File/File System level – Local Disk File System EDUCATIONSerial File System sharing on same or dissimilar OS via common Volume Manager & Physical File System Good for sequential (not concurrent) data processing and data migration If OS is different– Same Volume Manager avoids VTOC incompatibilities – Byte order differences may require meta-data conversion Intel is Little-endian, most others are Big EndianExamples– Homogeneous OS (common case) Most file systems (and volume managers) support this UFS, HFS, XFS, JFS, VxFS, ext2/3… SDS/SVM, LVM, XVM, VxVM… Sanbolic Kayo, DNF Dynamic Share– Heterogeneous OS (need common volume manager) VERITAS* Storage Foundation with Portable Data Container feature* Merged with Symantec, July 200522How is data shared ?File/File System level – Local Disk File SystemEDUCATIONExample: E lDW* ApplicationOS #0 OS #1 OS #2 OS #3- How ? OS #0 server stores d t t data OS #1 server starts batches OS #2 server loads data into the i t th DW OS #3 server backups data No data multiplication Cost effective for Storage More effective use of servers No time wasted in copying data between serversStorage NetworkImport Disk Group Start Volume Mount File System - Benefits Shared DisksDeport Disk Group Stop Volume Umount File System* Data Warehouse23How is data shared ?File/File System level – Cluster File SystemEDUCATION Cluster File System (also called Shared Data Cluster)– – – – A Cluster FS allow a FS and files to be shared All nodes understand Physical (on disk) FS structure (on-disk) The File System is mounted by all the nodes Single FS Image g g Same data view from all nodes– Examples HP CFS (TruCluster) HP/Cal. Soft. Monster FS IBM GPFS MACROIMPACT SANique CFS POLYSERVE Matrix Server REDHAT GFS 1 SANBOLIC MelioFS VERITAS2 CFS1 – Sistina acquired by RedHat 2 – Merged with Symantec, July 200524How is data shared ?File/File System level – Cluster FSWeb Server First HostCluster File System Cluster Volume ManagerEDUCATION Web Server Web Server Second HostHeartBeat Lock ManagementClusterStorage StExample:NetworkWeb Servers Farm How ? Shared VM/FS Load Balancer in front Increased throughput More effective use of servers Failure is transparent SSI/SFSI, High SLAsShared DisksBenefitsOptional Layer25How is data shared ?File/File System level – Cluster File System Asymmetric & Symmetric Implementation– Asymmetric uses master node for logging and lockingEDUCATION L k Mechanism Lock M h i– Distributed or Global Lock Management (DLM/GLM) Different implementation strategies – Granularity varies: file, record, byte… file record byte Cache Coherence – Single File System Image– E er modification is seen b all nodes as soon as a Every by modification in the data sharing domain occurs Usage Consideration: Concurrent vs Serial data access– Concurrent: multiple systems access the data simultaneously – Serial: one system at a time uses and access the data26How is data shared ?Advanced methods – File/FS ApproachesEDUCATION SAN File System– 1 node (Master) or a set of masters Understand, manage and use metadata on disk Use of file system even if portions of it are inaccessible block addresses distributed to nodes (clients) on request– Other nodes (clients) connection to SAN storage Avoid overhead due to Metadata management access to data directly using blocks addresses sent by Master(s)– Designed to support hundreds or thousands of nodes – Mixed role between direct data access with host based thin software and NAS access27How is data shared ?File/File System level – SAN File SystemEDUCATION– Flexibility of network FS at SAN speed y p – Long-term goal for the industry development for Capacity and Performance scaling Scaling hundreds of PetaBytes of capacity and tens of GigaBytes/sec – More recent File Server Generation – Examples p APPLE Xsan ADIC StorNext FS DataPlow SAN FS & Nasan FS EMC Celerra HighRoad, g , MPFS/MPFSi HP DirectNFS – xNFS (Cal. Soft.) – Transoft Fibrenet IBM TotalStorage SAN FS FS, SANergy IBRIX FusionFS SGI CXFS SUN QFS28How is data shared ?File/File System level – SAN FSApp. File Server Data Network - LANMetadata ServerEDUCATIONApp. File RequestApp.NFS/CIFS Server SClient swClient swBlock listData and Control A C l AccessStorage Network Data AccessExample:Shared DisksMultimedia A li ti M lti di Application How ? 1 big server with NFS/CIFS layer Server and Client SAN FS layer Hundreds f li t H d d of clients Increased throughput Consolidate storage, very scalable More effective use of resources M ff ti fBenefits 29How is data shared ?File/File System level – SAN FS H How it works ? k– Asymmetric or Client/Server model – Server controls client access, resolves conflicts – Thin client software layer handles SAN device and server interactionEDUCATION Lock Mechanism– – – – Provided by the server at a central location Various granularity: file, record, byte… g y , , y Some implementations use SMB or NFS semantics The server needs to be protected cause it represents a SPOF Cache Coherency– Some implementations deliver cache coherency with traditional validate/invalidate mechanism, others don’t offer cache at all30How is data shared ?File/File System level – OSD OSD = Object based Storage Device Object-based An object comprises– Application data (e.g., file, record) – Device-managed metadata (e.g., block allocation) – User-accessible attributes (e.g., access times)EDUCATION Objects have file-like methods for access– Open, close, read, write, get/set attributes – Commands are authorized Industry offerings y g– Lustre () – Bull, CFS, Cray, HP, Scali, SUN… – Lustre based – PanasasID x123 Blocks:3,42 Blocks:3 42 Length:512 SNIA OSD Working Group– OSD as a SCSI command set– /tech_activities/workgroups/osd g g pCheck out SNIA T t i l: Tutorial:Object-based Storage Device31How is data shared ?File/File System level – OSDCPUApplicationsSystem Call Interface File System User Component File System Storage Component Object Interface Block Interface File System Storage Component Block I/O Manager g Block I/O ManagerEDUCATIONCPUApplicationsSystem C ll Interface S t Call I t f File System User ComponentStorage DeviceStorage Device32How is data shared ?File/File System level – OSDClientsEDUCATIONEth switch SANSECRET KEYManagersIntelligent Device Space Management g Backup/Recovery QoS via attributes SecuritySECRET KEY SECRET KEYValidate Capability p yObject-based Storage Devices Obj t b d St D i33How is data shared ?File/File System level – Parallel NFS (pNFS)EDUCATIONNowClient Host NetClientpNFS GoalHost NetDataStorage NetNFS ServerStorage NetNFS ServerDataStorage ServersStorage Servers Allow NFS 4 d t t b All NFSv4, data to bypass NFS server– No application changes, similar management modelpNFS extensions to NFSv4 communicate data location to clients– Cli t access d t via Fib Ch Clients data i Fibre Channel, iSCSI OSD or even NFS l iSCSI, OSD,IETF standardization in progress34How is data shared ?File/File System level – SANFS vs CFS Characteristics Ch t i ti & FeaturesTolerance of Distance (between server and clients) # of nodes Heterogeneous OS Dedicated Meta-Data Server(s) Required Physical filesystem y g layout knowledgeEDUCATIONSAN FSImportant Hundreds Yes Yes, usually Metadata server only (clients may understand if same OS)Cluster FSLimited Dozens No No – cluster assigns functions to nodes All nodes (Cluster FS currently requires same OS)35How is data shared ?File/File System level – Parallel File SystemEDUCATION C Concept/Idea: Data is striped b t t/Id D t i t i d between servers (I/O nodes) Features– Cluster-wide consistent name space – User control for file striping across I/O nodes Asymmetric (master + slave servers + clients)– G GoogleFS, PVFS*, IBRIX, P l FS PVFS* IBRIX Panasas ( d) (osd) Symmetric (peer servers + clients)– TerraScale Isilon Exanet NetApp (Spinnaker Networks) TerraScale, Isilon, Exanet,* Parallel Virtual File System36How is data shared ?File/File System level – Parallel File System Asymmetric SymmetricEDUCATION#0 #1 #237EDUCATIONConclusion C l iConclusionVarious ways to Share Data…EDUCATION Many products and philosophy in the industry– – – – OS, disk (local) file system Methods to protect data (locking) Cache coherency mechanisms and semantics Caused by varied objectives and applications y j pp There is no single, simple, efficient data format available on all operating systems !! (sorry)– Server and client software needed for Data Sharing – Remember VTOC and Byte ordering potential issue39Conclusion… to leverage the infrastructureEDUCATION There are a number of things to consider when choosing a file system or server– – – – – Will th application work as d i d? the li ti k desired? Will it perform and scale? Does it have the required data management services? q g Is it secure enough? Is it easy to use and manage? There is no single solution that is superior in all cases BUT these approaches deliver real applications and business benefits– Real measured ROI – Performance Availability and Manageability Performance,40Q&A / FeedbackQ&A/FeedbackEDUCATION •Please send any questions or comments on this presentation to SNIA: track-storage@Many thanks to the following individualsfor their contributions to this tutorial.SNIA Education CommitteeSNIA Ed ti C itt •Symantec(Philippe Nicolas)•ONStor(Jonathan Goldick)•EMC²(David Black)•CA, Cisco, CNT, Crossroads, EvaluatorGroup, HDS, HGAI, Inrange, Knowledge Transfer, Microsoft, NationWide,QLogic, Sandia National Laboratories, Seagate,SolutionTechnology, Sun Microsystems & VERITAS Software41EDUCATIONAdvanced Data Sharing Technologiesversion 9Philippe Nicolas, Symantec, Jonathan Goldick, ONStor。
