QoS-aware Network Operating System for Software Defined Networking with Generalized OpenFlows

QoS-aware G Network Operating System for Software Defined Networking with Generalized OpenFlowsKwangtae Jeong, Jinwook Kim and Young-Tak KimDept. of Information and Communication Engineering, Graduate School, Yeungnam University, Korea Email: { kals25, jwkim}@ynu.ac.kr, ytkim@yu.ac.krAbstract — OpenFlow switching and Network Operating S ystem (NOX) have been proposed to support new conceptual networking trials for fine-grained control and visibility. The OpenFlow is expected to provide multi-layer networking with switching capability of Ethernet, MPLS , and IP routing. NOX provides logically centralized access to high-level network abstraction and exerts control over the network by installing flow entries in OpenFlow compatible switches. The NOX, however, is missing the necessary functions for QoS -guaranteed software defined networking (S DN) service provisioning on carrier grade provider Internet, such as QoS-aware virtual network embedding, end-to-end network QoS assessment, and collaborations among control elements in other domain network.In this paper, we propose a QoS -aware Network Operating S ystem (QNOX) for S DN with Generalized OpenFlows. The functional modules and operations of QNOX for QoS-aware SDN service provisioning with the major components (e.g., service element (SE), control element (CE), management element (ME), and cognitive knowledge element (CKE)) are explained in detail. The current status of prototype implementation and performances are explained. The scalability of the QNOX is also analyzed to confirm that the proposed framework can be applied for carrier grade large scale provider Internet 1.Keywords – Net work Operat ing Syst em, OpenFlow, GMPLS, Traffic Engineering, QoSI.I NTRODUCTIONNetwork operating system (NOX) enables management applications to be written as centralized programs over high-level abstractions of network resources as opposed to the distributed algorithms over low-level addresses [1, 2]. The network operating system does not manage the network itself; it provides a programming interface with high-level abstractions of network resources (e.g., CPU processing power, memory, disk storage volume, link capacity, etc.) that enable network application programs to carryout complicated tasks safely and efficiently on a wide heterogeneity of networking technologies [1]. The NOX, however, fails in providing the necessary functions for QoS-guaranteed so f tware de fined networking (SDN) [3] service provisioning on carrier grade provider Internet, such as QoS-aware virtual network embedding, end-to-end network QoS assessment, and collaborations among control elements in other domain network. The SDN is a networking paradigm which allows network operators to manage networking elements using software 1This research work was supported by Yeungnam University research grants in 2011 and HiMang Project (No 100035594) funded by MKE, Korea.running on an external server [3]. This is accomplished by asplit in the networking architecture between forwarding element and control element. Separation of control software from numerous packet forwarding nodes into a few centralized controllers has been proposed to increase flexibility in the deployments of new services (e.g., overlay networking, virtual private network, cloud computing, and content distributions), programmability with standardized open API, and reliability in the converged I P network [3-6]. The installation of control software in a few controller nodes remotely from the forwarding elements reduces the software complexity of numerous forwarding elements, and increases the overall reliability of the network [4]. Two technologies which allow the split of control element from forwarding element are ForCES (Forwarding and Control Element Separation) [7, 8] and OpenFlow [3].The ForCES working group in ETF is providing the framework and associated standardized protocol for information exchange between the control element and the forwarding element for open programmable networks [7-9]. Recently, the separation of control plane from the data forwarding plane has been also used in OpenFlow that is supporting flow-based forwarding and fine-grained per-flow controls, and it provides a new flexibility in the research work for future Internet architectures [10-12].Currently, OpenFlow features have been added on experimental basis to the HP ProCurve 5400 and CiscoCatalyst 6500 series switches [12]. OpenFlow switch implementation on the NetFPGA can provide I Pv4 packetprocessing for four 1 GgE Ethernet ports at 1 Gbps line-rate [12]. In order to be a practically applicable solution for a carrier grade large-scale I nternet, however, the networking with separation of control and forwarding (including OpenFlow architecture) should be able to fulfill following requirements: scalability, reliability, quality of service (QoS) and service management [3]. The scalability for carrier grade networking should include following aspects: i) scalability o fthe centralized control element that must handle routing and path calculation for the forwarding elements, ii) scalability of the flow table in the forwarding element (especially the transit core router), and iii) scalability of fault restoration at link or node failure in forwarding elements. In this paper, we propose a QoS-aware Network Operating System (QNOX) for Software Defined Networking with Generalized OpenFlows. The functional modules andoperations for QoS-aware SDN service provisioning with the major components (e.g., service element (SE), control element1167978-1-4673-0269-2/12/$31.00c2012IEEE(CE), management element (ME), and cognitive knowledge element (CKE)) are explained in detail. The current status of prototype implementation and performances are explained. The scalability of the QNOX is also analyzed to confirm that the proposed framework can be applied for carrier grade large scale provider Internet.The remainder of this paper is organized as follows. Section briefly introduces related work on SDN, separation of control plane and forwarding plane, and OpenFlow. In SectionI I I , we propose a QoS-aware Network Operating System(QNOX) for Software Defined Networking with Generalized OpenFlows. The functional modules and operations for QoS-aware SDN service provisioning with the major components are explained in detail. Section IV explains the current status of implementations, and Section V provides performance analysis. Finally, Section VI concludes this paper.II.R ELATED W ORKA.Software Defined Networking (SDN) and NetworkOperating System (NOX)The SDN allows network operators to manage networking elements using software running on an external server [3]. SDN provides abstraction at three areas of transport networks (distributed state, forwarding, and configuration) which are key to extract simplicity. This is accomplished by a split in the networking architecture between forwarding element (FE) and control element (CE). Separation of control software from numerous packet forwarding nodes into a few centralized controllers has been proposed to increase the flexibility in the deployments of new services (e.g., overlay networking, virtual private network, cloud computing, and content distributions), the programmability with standardized open AP, and the reliability in the converged IP network [3-6].Since the installation of centralized control software in a few controller nodes remotely from the distributed forwarding elements reduces the software complexity of numerous forwarding elements, and it also increases the overall reliability of the network [4]. The SDN makes the introduction of new vendor operating systems much easier. It allows users to create plug-ins for adding features to the control plane without having to change the underlying hardware, or to enhance the hardware without changing the control plane.The network operating system (NOX)does not manage the network itself; it provides a programming interface with high-level abstractions of network resources (e.g., CPU processing power, memory, disk storage volume, link capacity, etc.) that enable network application programs to carryout complicated tasks safely and efficiently on a wide heterogeneity of networking technologies. The NOX, however, fails in providing the necessary functions for QoS-guaranteed software def ined networking (SDN) [3] service provisioning on carrier grade provider I nternet, such as QoS-aware virtual network embedding, end-to-end network QoS assessment, and collaborations among control elements in other domain network.B.Separation of Control Plane and Forwarding PlaneThe most important feature of future I nternet routers is supporting open programmable networking that provides i) flexibility in new services provisioning, ii) open and modular interfaces of control plane, and iii) controllability and programmability of the network with high-level abstractions [4]. In order to be open programmable, the control plane and forwarding plane of currentInternet routers should be systematically separated by standardized interfaces.The control plane functions include i) routing for reliable advertisement of the network topology and available resources (e.g., OSPF and BGP for IP networking), ii) path computation, iii) signaling (e.g., RSVP-TE for MPLS/GMPLS networking), and iv) traffic engineering database (TEDB). The complexity of control plane has been continuously increased in order to support new features, such as virtual private network (VPN), cloud computing, contents distribution networking, fast fault tolerance, and mobility.Separation of control plane (e.g., OSPF of IP router) from forwarding elements (such as current I P router with packet classification and forwarding) has been proposed and analyzed in recent research work [1, 2]. The IETF ForCES (forwarding and control element separation) working group is providing the framework and associated standardized protocol for information exchange between the control element and the forwarding element for open programmable networks [2, 3].C.OpenFlowThe OpenFlow network consists of OpenFlow compliant switches and OpenFlow controllers with unmodified end-hosts [10-12]. Essentially, the OpenFlow separates the datapath over which packets flow, from the control path that manages the datapath elements.OpenFlow has been deployed at various academic institutions and research laboratories, and recently several research work provide analysis on the performance of OpenFlow framework as a viable solution for high-performance commercial networks, such as data center networks [2].The control plane is responsible for the initial establishment of every fine-grained flow by configuring the related switches along the path in the domain. As a result, the fine-grained per-flow control and global visibility in OpenFlow framework imposes an excessive overhead on the forwarding element [16]. In order to mitigate the scalability problem of fine-grained per-flow control, an efficient hierarchical flow aggregation and traffic engineering with multi-layer networking [17] should be implemented. Details of the multi-layered networking with GMPLS are explained in following subsection.D. Multi-layered Networking with GMPLSThe futureInternet forwarding elements will include various packet or frame switching capabilities for QoS-aware service provisioning and efficient traffic engineering with fast fault restoration. Multi-layered networking [17] defines generic data plane with GMPLS (generalized multi-protocol label switching). GMPLS includes switching capabilities of following layers: layer 3 packet switching/forwarding, layer 2.5 MPLS switching, layer 2 Ethernet switching, layer 1.5 SONET/SDH cross connect, layer 1 WDM lambda switching, and layer 0 fiber switching [17].The Generalized OpenFlow defined in this paper specifies that the forwarding elements are providing the GMPLS switching capabilities in distributed manner. Especially, the11682012IEEE/IFIP4th Workshop on Management of the Future Internet(ManFI)MPLS-TP is used for QoS-aware traffic engineering, while WDM lambda switching is used for long distance wide area networking. The Generalized OpenFlow is not requiring end-to-end homogeneous signaling, but supports various connection establishments in each domain network. The overall connectivity is controlled by PCE (path computation element) in the control element [18-22]. III.Q O S-AWARE NETWORK O PERATING S YSYTEM FOR SDN WITH G ENERALIZED O PEN FL OWA.Generalized OpenFlow NetworkingThe OpenFlow version 1.0 is providing fine-grained per-flow controls based on 10-tuples (Ethernet switch input port, source/destination MAC address, Ethernet type, VLAN ID, IPsource/destination address, I P protocol, TCP/UDPsource/destination port). In the recent OpenFlow specification,MPLS has been included; but GMPLS networkingtechnologies based on WDM optical lambda switching are not included. Since efficient and scalable traffic engineering is oneof the most important factors in the future Internet, multi-layernetworking with optical transport network (i.e., ASON(Automatically Switched Optical Network) /WDM(Wavelength Division Multiplexing)) is essential.I n this paper, we propose a Generalized OpenFlow thatincludes I P, MPLS-TP and WDM/ASON. MPLS-TP layer ismostly used for traffic engineering for QoS-guaranteeddifferentiated service provisioning, while WDM/ASON layer isused for long distance transit networking. DifferentiatedMPLS-TP TE-LSPs (traffic engineering label switched paths)are pre-planned to configure virtual overlay networks for each DiffServ class type (i.e., NCT (network control traffic), EF(expedited forwarding), AF (assured forwarding) 4, AF3, AF2,and AF1). Multiple QoS-guaranteed I P packet flows are aggregated into a designated MPLS-TP for required QoS-aware SDN services. Multiple MPLS-TPs are routed along the optical lambda path in WDM/ASON for long distance transit networking. When fast fault restorations (e.g., less than 50ms of protection switching time) is required, working TE-LSP and SRLG (shared risk link group)-disjoint backup TE-LSP are installed together with constraint-based shortest path first (CSPF) multi-domain routing.Generalized OpenFlow switch provides actions for MPLS-TP and WDM/Optical networking in GMPLS networking concept. I t provides label push, label pop and label swap operations at the ingress LER (label edge router), egress LER, and intermediate transit LSR (label switch router), respectively.B.QoS-aware Network Operating System (QNOX) for SDNService Provisioning with Generalized OpenFlows Figure 1 depicts the overall framework of the QoS-awareNetwork Operating System (QNOX) for SDN serviceprovisioning with Generalized OpenFlows . The majorfunctional components of SDN/GOFN are service element(SE), control element (CE), management element (ME), and cognitive knowledge element (CKE). The forwarding element(FE) is Generalized OpenFlow (GOF) compatible switchingnode. The transport domain networks (i.e., FEs) may becomposed of i) Generalized OpenFlow compatible networkscontrolled by remote controllers, and ii) legacyI P/MPLS/WDM transport network which are controlled andmanaged by their own subnetwork-dependent control elements(e.g., OSPF & BGP, RSVP-TE) and management elements(e.g., SNMP/I P, TMN for SONET/SDH). I n addition to theusual switching node and links, the Generalized OpenFlowNetwork (GOFN) also includes cloud computing servers, datastorage servers, and multicasting switching, and other server node with special feature for content delivery network (CDN)services.Figure 1. QoS-aware Network Operating System (QNOX) for SDN Serviec Provisioning with Generalized OpenFlows2012IEEE/IFIP 4th Workshop on Management of the Future Internet (ManFI)1169In the proposed QNOX for SDN/GOFN framework, clients request services using QoS-aware Open Virtual Network Programming Interf ace (QOVNPI). The service element (SE)receives service requests with attributes of the required computing and storage capacity, location of the users’ access points, required performance and QoS parameters, required fault restoration performance, and security level. The SLA (service level agreement) and SLS (service level specification) module checks and evaluates the availability of the network resources, and determines the guaranteed-QoS service provisioning. If the requested QoS level is not available, there may be some negotiations on the QoS level and performance parameters among SE and the user. Currently, the QOVNPI is under developments, based on XML and JSON (JavaScript Object Notation). The SE also includes the service life-cycle management for the accepted services, and QoE/QoS monitoring functional module.The control element (CE) handles the end-to-end session control with path establishments on each transport networks for connection-oriented services, and flow table updates using CE-FE interactions along the route of the generalized OpenFlow. The CE includes SIP (session initiation protocol)/SDP (sessiondescription protocol) module for end-to-end QoS provisioningof realtime multimedia conversational services, such as VoI P and multimedia conference call. In order to provide flexibility of interworking between GOFN and legacy networks (such as P/SONET, I P/WiFi, and I P/GMPLS), the control plane is composed of transport network-independent control (including SI P/SDP, path computation element (PCE), and policy-based routing) and transport network-dependent control (including BGP, OSPF-I S I S BGP/I P, and RSVP-TE/MPLS). The Generalized OpenFlow control provides controller functions for Generalized OpenFlow compatible switches, such as creation or deletion of a GOF table entry. Since one subnetwork-dependent control node is allocated for each transport network domain, the subnetwork-independent control node (which handles end-to-end session connectivity) may interact with multiple subnetwork-dependent control element and/or Generalized OpenFlow control element.The management element (ME) performs network resource discovery, multi-layer/multi-domain QoS-aware virtual overlay networking, and virtual network topology managements. The ME is also composed of subnetwork-independent management function, subnetwork-dependent management function (such as SNMP for I P network, LMP/OAM for ASON, and TMN for SONET/SDH transmission network), and Generalized OpenFlow management function for GOF-compatible GMPLS switch nodes. The ME also provides network QoS performance monitoring.The cognitive knowledge element (CKE) maintains link state database (LSDB) and traffic engineering database (TEDB) of transport network topology, and provides the decision making of mapping a virtual network topology for the SDN user’s requested virtual topology onto the physical transport network topology. The CKE also supports traffic engineering for QoS-guaranteed service provisioning and network load balancing.C.ForCES Protocol for CE-FEIn the proposed QNOX for SDN/GOFN, the CE and the FE are interconnected by ForCES protocol [15], as shown inFigure 2, which provides much more features than OpenFlow SSL (secure sockets layer) secure channel. The ForCES interface is composed of two parts: the protocol layer (PL) and the transport mapping layer (TML). The PL is in fact the ForCES protocol that defines all the semantics and the message formats, while the TML is used to connect two ForCES PL entities on the CE and the FE, respectively.We implemented SCTP (stream control transport protocol)-based TML that provides 3 priority levels, according to RFC 5811 [16]. The high priority TML channel is used for association setup, association setup response, association teardown, configuration, configuration response, query, query response, and flow table entry update. The medium priority channel is used for event notification, while the low priority channel is used for packet redirect and heartbeat.Since one CE controls multiple FEs in the domain network, the CE that is connected with FEs using ForCES may become bottleneck in calculation of updated routes and downloading FI B table entries to each FE. The performance of CE-FE message exchanges using ForCES are analyzed in Section V.Control Element(CE)Forwarding Element(FE)ForCES ProtocolForCES Interface (PL/TML)ForCES Interface (PL/TML)Longest Prefix Matching (LPM) Forwarding Packet ClassificationMetering &MarkingTraffic shaperPer-class-typequeuingNetwork Address Translation Multi-layer Path Computation Element (PCE)Routing Protocols (OSPF , BGP)Signaling Protocols (RSVP-TE, CR-LDP)Session Layer Signaling (SIP/SDP)Link/Node Fault DetectionLink/NodePerformanceMonitoring Management Element(ME)User defined controlapplications CE Manager FE Manager Virtual Network Topology Manager TE DB Link StateDBPerformance MonitoringNetwork Load BalancingKnowledge-based Decision making on service & networkmapping FE ManagementAgentCE ManagementAgentCognitive KnowledgeElement (CKE)Network resource managerFigure 2. Functional Blocks in QNOX for SDN/GOFNwork Resource Discovery by MEWhen an FE is activated and initialized, it firstly connects to the ME based on the designated ME’s I P address for the subnetwork domain network, via its neighbor FE node. We are implementing UPnP-UP (universal plug and play with user profile) [21-22] -based protocol that forwards the association setup request message from an FE to the ME/CE by the neighbor FE node(s).When the newly initialized FE is connected to the ME through its neighbor, the ME establishes a secure connection with the FE, discovers the available network resources of the FE (e.g., number of interfaces and their capacities), provides additional information for the initialization (including the CE address of the FE). The FE then makes an association with the designated CE; after obtaining the information of network interfaces, the CE updates forwarding information base (FI B) for all existing FEs, and downloads the updated FIB to each FE, including the newly added one. The ME also collects the information on the links of the newly added FE, updates the network topology, and TEDB.11702012IEEE/IFIP 4th Workshop on Management of the Future Internet (ManFI)E.Calculation of inner routes and outer routes for each FEs The interfaces of an FE are connected to either internal link or external link, as shown in Figure 3. When the CE calculates and installs the forwarding information base (FIB) for each FE, there are five steps: i) calculate inner routes among FEs within the domain network controlled by the CE, ii) calculate outer routes that connect with outside external router or other domain network controlled by other CE, iii) merge the FI B table for each FE, iv) compare with its old version to find any updated FIB table entry, and v) download and install only the updated FIB table entries for individual FE.Firstly the CE calculates the inner routes for each active external interface from each FE by configuration of shorted path first spanning tree. When a link or network interface failure is occurred at any FE, the CE also updates the spanning trees, excluding the failed interface and link. If the CE receives any link state advertisement (LSA) from a CE in other domain network or external router, the routes to the connecting external interface should be calculated or updated for every FE individually. The overall performance of the fault notification from FE to CE, route calculation and F B update at CE, selection and downloading FIB entries for individual FE from CE, and installation of the updated FIB entries at FE, should be limited to be less than 50 ms in order to guarantee the QoS-provisioning of carrier grade networks.FE1_1FE1_2FE1_3FE1_4CE1FE2_1FE2_2FE2_3FE2_3CE2Figure 3. Sample network topology with CE and FEsWhen the virtual network topology of the requested serviceshould be mapped across multiple domain networks which are controlled by different CEs, the path computation element (PCE) [18-22] of each domain network collaborates to find the optimum routes from the ingress node to the egress node using backward recursive PCE-based path computation, as depicted in Figure 4. The CE of the source domain network sends the QoS-aware data path setup request to the destination domain network, through all possible routes. The CE of the destination domain network then calculates constraint-based shortest paths from the entry boundary nodes (BNen) to the destination egress node. I t then delivers the computed accumulated cost from each BNen to the PCEs of its upstream domain network, using PCEP (PCE protocol) [19] response message. The PCEs of the upstream domain networks calculate the accumulated cost of constraint-based shortest path from its entry boundary nodes to the destination egress node, through the selected boundary entry nodes in downstream domain networks. This procedurerepeats in all domain networks, and the PCE of the source domain network can select the shortest path from the multiple candidates of path delivered by the PCEP response messages through different routes.Figure 4. Backward Recursive PCE-based Path Computation across MultipleDomain NetworksF.Multi-layer Overlay NetworkingFor better scalability in QoS-aware traffic engineering and fault restoration, the proposed Generalized OpenFlow Network (GOFN) provides multi-layer virtual networking, as shown in Figure 5. Basically, two overlay networks are configured as preplanned infrastructure: (i) WDM/ASON layer network, and (ii) MPLS-TP layer network. MPLS-TP virtual networks are mostly used for QoS-aware traffic engineering and fast fault restoration, while WDM/ASON virtual networks are used for long distance transit networking with SRLG-disjoint working path and backup path configurations.Figure 5. Multi-layer Virtual NetworkingThe three virtual layer networks shown in Figure 5 are configured in client-server relationship. The CE of each domain network includes the functional modules of PCE (path computation element) and CC (connection controller) for each layer network. The CC is used to invoke any available signaling in the layer network, or to create a new Generalized OpenFlow in the GOF compatible switching nodes. The CKE maintains the up-to-date TEDB (traffic engineering database) of each layer network, while ME maintains the overall network2012IEEE/IFIP 4th Workshop on Management of the Future Internet (ManFI)1171status with the VNTM (virtual network topology manager). The TEDB is used in the policy-based routing in each overlay network, where the routing policy may be usual shortest path first, load balancing, equal cost multi-path (ECMP), or energy saving.The virtual overlay network for the requested SDN service is configured on the MPLS-TP virtual networks that provide appropriate QoS and fault restoration. As explained in Figure 1, the mapping between the virtual network topology of the requested QoS-aware SDN service and the virtual network topology of the provider’s GOFN topology is processed at the cognitive knowledge element (CKE).G.Virtual Network EmbeddingI n the QoS-aware SDN service request, a virtual network topology with a collection of virtual nodes and virtual links are defined, and this virtual network should be mapped onto the substrate network (i.e., provider’s network). The virtual nodes are defined with attributes of requested cloud computing CPU power, amount of memory size, amount of storage, and any specific features on the node for content delivery network services. The virtual links are specified with attributes of required bandwidth, QoS (delay, jitter, packet loss, packet error) of the link and path. The fault restoration performance of each link and path is specified with performance parameters of fault restoration time, SRLG-aware working path and backup path installation, and protection type (e.g., 1:1, 1+1, 1:N, or M:N).The most important SDN services in future I nternet are cloud computing service and content delivery service, and they must be provided for a user group in a certain location, the allowed distance (or delay) from the users and the server nodes may be defined.In the proposed QoS-aware QNOX for SDN/GOFN architecture, the CKE handles the virtual network mapping for any SDN service request. Currently, a simple mapping algorithm is implemented in CKE. Study on a more sophisticated and scalable mapping algorithm for realistic cloud computing and content delivery services with practical constraints is under development.H.End-to-End QoS Monitoring and ManagementContinuous monitoring of the end-to-end network QoS and assessment on the guaranteed QoS provisioning according to SLA/SLS are very important in the future I nternet. I n the proposed framework of the QNOX for SDN/GOFN, the network QoS monitoring functions are implemented at the ingress node and the egress node of the QoS-aware path shown in Figure 4. The parameters of the QoS monitoring include delay, jitter, packet loss, and packet error.Since MPLS-TP overlay networks are used for differentiated service provisioning, MPLS OAM (operation, administration, and maintenance) performance management functions are used to monitor the domain network QoS between the entry boundary node (BNen) and the exit boundary node (BNex), and the end-to-end network QoS between the ingress node and the edges node, as shown in Figure 4.The monitored results of the domain network QoS and the end-to-end QoS are sent to the CKE and stored in the knowledge base and TEDB. The CKE analyzes the overall QoS provisioning of the GOFN for QoS-aware SDN services and adjust the operational parameters of traffic engineering (e.g., policy of buffer management at each OpenFlow switch, link utilization level, and policy on routing).IV.I MPLEMENTATION OF Q O S-AWARE N ETWORKO PERATING S YSTEM FOR G ENERALIZED O PEN F LOWN ETWORKSA.Generalized OpenFlow Forwarding Element (FE) andControl Element (CE)The Linux I P/MPLS router has been modified to emulate the proposed GOFN forwarding element (FE) and control element (CE). The OSPF routing module is separated from the I P/MPLS router packet forwarding, and implemented in the centralized CE. One CE is configured for a domain network of GOFN which contains 2 ~ 114 FEs. Basic I P packet routing and MPLS frame switching functions are included in the FE. Routing I nformation Base (RIB) configured by BGP/OSPF is maintained in the CE, while Forwarding I nformation Base (F B) is calculated for each FE individually by the CE, downloaded to each FE. Figure 6 depicts the functional architecture of CE and FE based on Linux IP/MPLS router.The proposed architecture was tested with various test network topology with upto 114 FEs which are controlled and managed by 1 CE and 1 ME. Each 19 FEs are grouped and installed on a PC server (I ntel(R) Xeon(TM) 3.20GHz, 140 GByte HDD, 12 GByte RAM, WinXP-64bit) where each FE is individually running on a virtual machine provided by VMware on Windows Server OS. One of the 19 FEs in the group is providing a direct link to CE that is shared by the 19 FEs.Each FE has a separated ForCES TML connection with CE for the control message exchange. The CE and The ME are configured to be running on a separated virtual machine.Figure 6. Functional architecture of CE and FE based on Linux IP/MPLSopen source11722012IEEE/IFIP4th Workshop on Management of the Future Internet(ManFI)。