One way to support OpenMP in a distributed memory environment is to use a software distributed shared memory system (SDSM) as an underlying runtime system for OpenMP. Cluster-enabled Omni OpenMP on SCASH is an implementation of Omni OpenMP compiler for a software distributed shared memory system SCASH running under SCore Cluster System Software.
In the most SDSMs, only part of the address space is shared. In SCASH, the address space allocated by a shared memory allocation primitive can be shared among the processors. Variables declared in global scope are private in the processor. We call this memory model, ``shmem memory model''. Parallel programs using the Unix ``shmem'' system calls use this memory model. In this model, all shared variables must be allocated at run-time at the beginning of execution. We have implemented the OpenMP compiler for shmem memory model using Omni OpenMP compiler system. The compiler detects references to a shared data object, and rewrite it to the reference into objects re-allocated in shared memory area.
SCASH is a page-based software distributed shared memory system using the PM low-latency and high bandwidth communication library PM for a Myrinet gigabit network and memory management functions, such as memory protection, supported by an operating system kernel. The consistency of shared memory is maintained on a per-page basis. SCASH supports two page consistency protocols, invalidate and update . The home node of a page is a node that keeps the latest data of the page. In the invalidate protocol, the home node sends an invalidation message to nodes which shares the page so that the page is invalidated on these node. The invalidate protocol is used as default.
SCASH is based on the Release Consistency (RC) memory model with the multiple writers protocol. The consistency of a shared memory area is maintained at a synchronization called the "memory barrier synchronization" point such as a barrier. At this point, only modified part is transferred to update pages. Explicit consistency management primitives are also supported for the lock operations.
double x; /* global variable declaration */ double a; /* global array declaration */ ... a = x;is transformed as follows:
double *_G_x; /* indirect pointer to "x" */ double *_G_a; /* indirect pointer to "a" */ ... (_G_a) = (*_G_x); /* reference through the pointers */The compiler generates the global data initialization function for each compilation unit. The initialization functions are linked and called at the beginning of execution to allocate the object in shared address space and store the address into the indirect pointer.
We have implemented an OpenMP compiler for a full set of OpenMP on the software distributed shared memory system, SCASH. We found that the page home distribution is a key to achieve the good performance on the SDSM through our several experiments on RWCP PC clusters.
Currently, we are introducing OpenMP extensions to specify data mapping in a flexible way, which gives application specific knowledge to the compiler. The loop scheduling to exploit locality for data mapping is also an important issue to investigate. Recently, we extended the directives to specifiy the data mappging and loop scheudling. We expect that these extensions improve the performance of OpenMP programs on SCASH.