iMeshP.h -- ITAPS Parallel Mesh Interface
Release 0.1; October 2008
- The term "mesh" refers to an abstraction in the data model; it does not imply a serial or parallel distribution.
- The term "partition" refers to an assignment of a set of entities to subsets; like a "mesh," it does not imply a serial or parallel implementation.
- An application may use one or more meshes.
- Parititions can create subsets of entities from one or more meshes.
- Meshes can be subdivided by one or more partitions.
- Partitions contain parts. Parts contain the subsets of entities in the partition.
- A "process" can be thought of as an MPI process. The number of processes can be considered to be the result of MPI_Comm_size. The rank of a process can be thought of as the result of MPI_Comm_rank. We will think in terms of processes rather than processors. Initial implementations of the parallel interface will likely use MPI terminology directly; future implementations may accommodate other communication paradigms and libraries.
- Partitions have communicators associated with them. These communicators can be thought of as MPI communicators.
- "Global" operations are operations performed with respect to a partition's communicator.
- "Local" operations are operations performed with respect to a part or a mesh instance within a process.
- Part A "neighbors" Part B if Part A has copies of entities owned by Part B and/or if Part B has copies of entities owned by Part A.
- Each process has one or more "mesh instances." A mesh instance can be thought of as a mesh database. An implementation should support the existence of more than one mesh instance per process (e.g., it should always associate mesh data with a mesh instance). However, we expect applications would most often use only one mesh instance per process.
- There is one root set per mesh instance.
- Each process may have one or more partition handles.
- A partition assigns entities from one mesh instance to parts.
- Entities in a mesh instance can be partitioned by one or more partitions. Mesh instances know which partitions they contain.
- Parts are uniquely identified globally by part IDs of type iMeshP_Part. Local parts can also be accessed by part handles that provide more direct access to a part. Functions accepting part handles operate correctly on only local parts (parts on the calling process); they will return an error for remote (off-process) parts.
- Generation and management of global IDs for entities is not included in the iMeshP interface. It can be provided as a service above the iMeshP interface. Uniqueness of global IDs is managed at the partition level.
- Each part is wholly contained within a process.
- A process may have zero, one or multiple parts.
- For each entity that is copied onto remote parts, the owning part knows both the remote part ID and remote entity handle of all copies.
- All parts with copies of a boundary entity know the remote part ID and remote entity handle of all copies of the entity.
- All parts with copies of any entity know the part ID and entity handle corresponding to the owner of the entity.
- Functions that return entity information for a part, set or mesh instance return the information for all entities (including copies and ghosts) in that part, set or mesh instance. Applications can check whether an entity is owned or a ghost using iMeshP_isEntOwner or iMeshP_getEntStatus.
- Many iMesh functions that accept an iBase_EntitySetHandle are also useful in the context of a iMeshP_PartHandle. These functions are reinterpreted so that they can accept either an iBase_EntitySetHandle or an iMeshP_PartHandle.
- In particular, entities are added to and removed from local parts via the same functions that are used to manipulate entity sets. That is, given a mesh instance, an entity handle, and a part handle, the entity is added to or removed from the part via calls to the following functions with the part handle passed as the entity set handle:
- Add entity to part --> iMesh_addEntToSet
- Remove entity from part --> iMesh_rmvEntFromSet
- Add array of entities to part --> iMesh_addEntArrToSet
- Remove array of entities from part --> iMesh_rmvEntArrFromSet
- Each function description includes its communication requirements. The options are described here:
- COMMUNICATION: Collective -- the function must be called by all processes in the partition's communicator. (These functions have the suffix "All" to indicate collective communication is done.)
- COMMUNICATION: Point-to-Point -- communication is used, but the communication is from one process to only one other process. The receiving process must issue an appropriate receive call to receive the message.
- COMMUNICATION: None -- the function does not use communication; only local operations are performed.
- COMMUNICATION: None++ -- no communication is done; the values are precomputed by iMeshP_syncPartitionAll or iMeshP_syncMeshAll.
- Non-blocking calls for off-processor mesh-modification return a request that indicates whether or not the operation has completed. The request is more than an MPI request; it encapsulates both the MPI information and the mesh operations that were requested. If non-blocking calls are used, appropriate calls to iMeshP "wait" or "poll" functions must be used to handle and satisfy requests.
