ITAPS

IPComMan Service

The Inter-Processor Communication Manager (IPComMan) [1] is a general-purpose communication package built on top of MPI that aims to reduce data exchange costs by exploiting communications of a local neighborhood for each processor. The neighborhood is the subset of processors exchanging messages with each other during a specific communication round, which in a number of applications is bounded by a constant, independent of the total number of processors. The communication package keeps the message-passing within subdomains, and eliminates, or reduces, the number of collective calls needed.

The communication package takes care of the message flow with a subset of MPI functions. IPComMan takes advantage of non-blocking message-passing functions, allowing it to post send or receive requests in between computation steps, and automatically manages their completion and delivery. The package provides several useful features: i) automatic message packing, ii) management of sends and receives with non-blocking MPI functions, iii) communication pattern reusability, iv) asynchronous behavior unless the other is specified, and v) support of dynamically changing neighborhoods during communication steps.

Each processor has its own neighborhood, which is different from that of its neighbor(s). While initializing an IPComMan object, each processor specifies the neighbors it is going to communicate with. From that point, the communication package is concentrated on delivering messages between the processor and its neighbors only, not touching other processors of the domain, when possible. There are no collective calls during each communication round. However, there could be situations when it is not possible to a priori define all the neighbors for processors, i.e., new neighbors may be encountered during a communication step. For example, mesh modification operations [2] may alter the neighborhood of specific processors. As new neighbors needing to communicate will be unaware of each other, one collective call at the end of communication round is performed to verify whether the global number of sends and receives match.

IPComMan is currently used to support parallel communication in FMDB - ITAPS interface implementation, and Mesh Adapt ITAPS Service. To measure the performance of mesh migration [3], an essential part of parallel unstructured mesh adaptation, IPComMan was compared with the Autopack communication utility [4] in three examples. The first two examples consider mesh size fields that represent a planar shock on cube geometry (CUBE) and two spherical shocks on torus geometry (TORUS). The third example includes mesh size field that represents the motion of air bubbles in a steady uniform flow (BUBBLE). These tests involve substantial coarsening and swapping and associated with them mesh migration.

parallel isotropic mesh

Figure 1. Parallel isotropic mesh adaptation on cube geometry

TORUS test - initial uniform mesh

Figure 2. Parallel anisotropic mesh adaptation on torus geometry.

parallel mesh adaptation for moving air bubbles

Figure 3. Parallel mesh adaptation for moving air bubbles.

Figure 1 shows the CUBE test with the initial uniform mesh of about 17 million tetrahedra, and the final mesh of 2 million tetrahedra. Figure 2 shows TORUS test where the initial uniform mesh consists of 68 million tetrahedra, and the final one contains 3 million tetrahedra. Figure 3 represents BUBBLE test, involving movement of five air bubbles by a distance of 1/5th of their radius, with the initial mesh having 165 million tetrahedra, and 188 million tetrahedra in the final mesh. Table 1 shows the communication time (on processor with maximum) needed for exchanging/migrating the entities during mesh adaptation routines. It also contains the ratio (labeled as Ratio) between communication time and overall time. The results on IBM Blue Gene/L show that IPComMan's ability to localize communication to neighborhoods to be independent of the total number of processors and its use of non-blocking functions allows it to continuously reduce the communication time as the number of processors increases. Even though the differences in communication times between Autopack and IPComMan are not substantial at 128 processors, IPComMan is from 3 to 7 times faster in the 4096 processor case. IPComMan was successfully run on up to 32k cores on IBM Blue Gene/P as a part of Mesh Adapt Service testing.

Test case

N/proc

128

256

512

1024

2048

4096

CUBE

Autopack

90

75

73

82

96

142

Scaling

1

0.60

0.31

0.14

0.06

0.02

Ratio

0.32

0.39

0.46

0.55

0.63

0.71

IPComMan

75

41

34.5

32.5

26.5

21

Scaling

1

0.91

0.54

0.29

0.18

0.11

Ratio

0.29

0.26

0.30

0.33

0.34

0.31

TORUS

Autopack

385

291

197

117

125

146

Scaling

1

0.66

0.49

0.41

0.19

0.08

Ratio

0.39

0.40

0.49

0.51

0.62

0.73

IPComMan

310

166

93

50

37

25

Scaling

1

0.93

0.83

0.78

0.52

0.39

Ratio

0.34

0.28

0.3

0.31

0.34

0.34

BUBBLE

Autopack

 

 

 

116

99

102

Scaling

 

 

 

1

0.59

0.28

Ratio

 

 

 

0.31

0.39

0.51

IPComMan

 

 

 

75

64

41

Scaling

 

 

 

1

0.59

0.46

Ratio

 

 

 

0.22

0.28

0.31

Table 1. Communication time (maximum) spent on mesh migration routines during mesh adaptation for the same tests presented in Table 1.

References

[1] A. Ovcharenko, O. Sahni, K.E. Jansen, C.D. Carothers and M.S. Shephard, Neighborhood communication paradigm to increase scalability in large-scale dynamic scientific applications. Parallel Comput., under review, 2010.

[2] F. Alauzet, X. Li, S. Seol, and M.S. Shephard, Parallel anisotropic 3D mesh adaptation by mesh modification, Engng. Comput., 21(3):247-258, 2006.

[3] E.S. Seol, M.S. Shephard, Efficient distributed mesh data structure for parallel automated adaptive analysis, Eng. Comput., 22: 197-213, 2006.

[4] R. Loy, Autopack user manual, Science Division Argonne National Laboratory,
http://www.mcs.anl.gov/research/projects/autopack/autopack_manual.pdf.

 

POC

Aleksandr Ovcharenko, Mark S. Shephard
Email: {shurik@scorec.rpi.edu, shephard@scorec.rpi.edu}

Funding sources

NSF Grant No. 0749152, DOE Grant No. DE-FC02-06ER25769, RPI CCNI

Download

The repository download is accessible at the following URL:
http://redmine.scorec.rpi.edu/anonsvn/ipcomman/trunk/