The Effectiveness of SRAM Network Caches in Clustered DSMs

The Effectiveness of SRAM Network Caches in Clustered DSMs

Adrian Moga and Michel Dubois

Department of Electrical Engineering - Systems

University of Southern California

Los Angeles, CA 90089-2562

(213)740-4475

Fax: (213) 740-7290

{moga,dubois}@paris.usc.edu

Abstract

The frequency of accesses to remote data is a key factor

affecting the performance of all Distributed Shared Memory(DSM) systems. Remote data caching is one of the most effective

and general techniques to fight processor stalls due to remote

capacity misses in the processor caches. The design space ofremote data caches (RDC) has many dimensions and one essen-

tial performance trade-off: hit ratio versus speed. Some recent

commercial systems have opted for large and slow (S)DRAM net-work caches (NC), but others completely avoid them because of

their damaging effects on the remote/local latency ratio.

In this paper we will explore small and fast SRAM networkcaches as a means to reduce the remote stalls and capacity traf-

fic of multiprocessor clusters. The major appeal of SRAM NCs is

that they add less penalty on the latency of NC hits and remoteaccesses. Their small capacity can handle conflict misses and a

limited amount of capacity misses. However, they can be coupled

with main memory page caches which satisfy the bulk of capacitymisses. To maximize performance for a large spectrum of appli-

cations, we propose to organize the NC as a victim cache for

remote data. We also propose a novel and scalable method tocontrol the page cache by integrating page relocation mecha-

nisms into the network victim cache.

1. Introduction

In a DSM system the latency of remote data accesses can befrom several times to an order of magnitude higher than that of

local data accesses. Processor caches capitalize on the temporal

locality of memory accesses and significantly alleviate the highlatency problem. However, speed and/or cost considerations

limit their size. Consequently, in large applications, the remote

working set overflows the cache, and capacity misses to remotedata generate large access penalties.

Several techniques have been applied at various system levels

to cut the number of remote capacity misses. Techniques avail-able to the programmer and compiler, such as blocking [11], and

Funding for this work has been provided by the National Science Foundation under Grants MIP-9223812 and MIP-9633542.to the operating system, such as page placement [13] migrationand replication [22][23], are effective up to a point. Further

improvements are attained by caching remote data in local mem-

ory resources acting as backups for the processor caches. Thismore powerful approach eliminates the hassle of dealing with the

problem in software. One way or another, practically all com-

mercially available (KSR-1 [9], NUMA-Q [16], Exemplar [1]),prototype (DASH [15], S3.mp [18]), or proposed (DDM [5],

COMA-F [6], Simple COMA [21], R-NUMA [3]) DSM archi-

tectures have provisions for remote data caching. Notable excep-tions are the SGI Origin [14] and the HAL S1 [24] which rely

exclusively on page migration and replication.

The design of a cache for remote data involves several keydecisions. A fundamental choice is whether to allocate separate

resources for caching local and remote data or to cache both in

the same memory, as done in COMA [5][6]. A study [26] hasfound a slight performance advantage for systems with large ded-

icated remote data caches (RDCs). In this paper, we focus on

dedicated remote data caches (RDCs). The size of the RDCimpacts the choice of the memory technology (DRAM or

SRAM) and its speed. There is a clear performance trade-off

between the effect on the number of remote capacity misses(dependent on the RDC size) and the effect on remote access

latencies (dependent on the speed). The RDC can be organized as

a network cache [15][18][16][1] or as a page cache [21][20]managed under various policies. Finally, the bus protocol in a

multiprocessor cluster also affects the RDC design.

The contribution of this paper is to propose a design and eval-uate the performance of small and fast network caches (NCs) in

the context of clustered DSMs. The small NCs are built in the

same SRAM technology as the processor caches and should beconvenient to use. We present and evaluate several techniques

that allow a cluster to achieve the same low remote capacity miss

ratio as if equipped with a much larger NC, but without payingthe penalty of slower remote accesses. Essentially, we propose to

organize the NC as a victim cache for remote data and to expand

its capacity with a page cache in main memory. We also proposea novel mechanism to control the page cache using the NC. Our