Real-Time Message Passing Specification based on MPI
(MPI is the Message-Passing Interface)

This is the home page of the MPI/RT Forum. Look for regular updates,
improvements, new papers and drafts, as well as announcements of interest.

The final approved version of the MPI/RT 1.1 standard is now available in Postscript (PS) and Acrobat (PDF) formats.

The previously approved MPI/RT 1.0 standard is also available in Postscript (PS) and Acrobat (PDF) formats.

For more information on astronomy, the moon and horoscope, see here :

MPI/RT Forum meetings are open meetings, and all interested individuals, and organizations may participate by e-mail, and/or by attending meetings of the Forum. All attendees at meetings participate in informal (straw) votes, and institutions have 1 vote on official decisions if they attend two of the last three meetings, including the one then currently in session.

Here are key goals of the MPI/RT Forum.


MPI/RT 1.1 Topics

Following is a list of topics under consideration for MPI/RT 1.1:

  1. Data-reorganization support
  2. Collective operations (ALL_TO_ALL)
  3. Dynamic Process Management, Parallel Client Server, etc.
  4. Interoperation of MPI/RT Implementations
  5. Support for Java binding of MPI/RT (other languages too?)
  6. Full-QOS-based MPI/RT with mode changes (such as a QOS capability for mode changes)
  7. Explicit ability to work with real-time schedulers for process scheduling
  8. Work to support k-slack channel mode (whereas MPI/RT 1.0 is 1-slack)
  9. Support for an IDL for MPI/RT
  10. Fine grain, Coarse grain QOS specifications
  11. More powerful data descriptors to allow strided and non contiguous buffers
  12. How to handle other resources’ schedulers. Currently we have dealt with memory and network in two different ways and via completely different specs (related to 7 and 10)
  13. Operations on containers. The old issues of propagating the operations to every object in a container (probably for the containers that contain homogeneous objects only)
  14. Container Iterators
  16. Channel state transition (old busy state)
  18. QoS Spec for channel (remove restrictions that were made for MPI/RT-1.0 at the last meeting)
  19. Default constructors
  20. New instrumentation metrics (Add as needed with experience)
  21. More error returns
  22. Remove restriction of homogeneity on bufiter parameters: dataspec and bufsize for channel endpoints
  23. Unreliable channels
  24. Fault tolerance, Fault handling, Fault recovery
  25. Light-weight handlers
  26. Half channels (I/O)
  27. Default handlers
  28. Group parameter scope for the commit
  29. Mechanisms for passing in externally build schedules to MPI/RT
  30. Logical grouping of event kinds (arithmetic on events)
  31. PGCOs
  32. QoS for handlers (add deadlines)
  33. Mutable buffers (late binding of address)
  34. Variable length buffers
  35. Periodic triggers
  36. Unique ID for running MPI/RT version
  37. QoS for QoS (granularity for raising errors)
  39. Operations on groups.

MPI/RT Forum Meetings

Dates Place Local Contact
June 5-6 MITRE, Boston Arkady Kanevsky
July 24-25 MITRE, Boston Arkady Kanevsky
September 25-26 MSU Anthony Skjellum
November 6-7 MITRE, Boston Arkady Kanevsky
December 11-12 SPAWAR, San Diego Dennis Cottel
January 14-16 Sandia, Albuquerque Ron Brightwell

February 24-26
SPAWAR, San Diego | Dennis Cottel
April 8-10
MITRE, Colorado Springs, CO | Alan Skillicorn
May 19-21 | Boston[MITRE] | Arkady Kanevsky
June 23-25
Mississippi State University, Starkville, MS | Tony Skjellum
July 22-24
DARPA - STA in Arlington, VA | Jon Hiller
December 2-4 | Boston[MITRE] | Arkady Kanevsky
February 3-5, 2024
Details | San Diego [SPAWAR] | Dennis Cottel
April 7-8, 2024
Details | Boston [MITRE] | Ken Cain
June 8-10, 2024
Details | Moorstown, NJ [Lockheed Martin - GES] | Nathan Doss
September 8-10, 2024
Hotels and Directions | Chelmsford, MA [SKY Computers, Inc.] | Steve Paavola
September 24, 2024
Chelmsford, MA [Mercury Computer Systems] | Arkady Kanevsky
 December 14-15, 2024
Details | San Diego [SPAWAR] | Dennis Cottel

MPI/RT 1.1 Forum Minutes

Minutes of the MPI/RT meeting Held on December 13-14, 2023 in Orlando, Florida


Murali Beddhu MPI Software Technology, Inc.
Dan Campbell GTRI
Dennis Cottel SPAWAR
Zhenqian Cui MPI Software Technology, Inc.
Nathan Doss LM-NESS
Arkady Kanevsky Mercury
Anthony Skjellum Mississippi State University

Five official votes (MSTI, MSU, Mercury, SPAWAR, LM-NESS)

September minutes approved after deleting item 3.

Arkady asked that proposals have dates.

  1. Proposal describing the meaning of formal votes for MPI/RT 1.1.
    FV-1: Approve the standalone proposal as is. If there are any
    major changes, FV-1 must be repeated.
    FV-2: Permission to integrate, with presumption of no more changes,
    unless some problem is discovered.
    FV-3: As part of finalization of the whole document, accept the
    integrated proposal, as amended to remove errors and to add

a. Proposal formally approved 4/0/0

  1. Errors Proposal v7.
    a. Item 45 clarified by Dennis; output cases removed.
    b. Item 1-42: FV-2, approved 4/0/0.
    c. Item 43-57: FV-1, approved 4/0/0.
    d. Items following 57 must be proposed at the next meeting to meet the
    voting schedule approved for MPI/RT 1.1 (see item 1 above).

  2. Error String Proposal
    a. No changes since September.
    b. FV-2, approved 4/0/0.

  3. Sleep Proposal
    a. Minor wording amendments were offered, not changing syntax or semantics.
    b. FV-2, approved 4/0/0.

  4. Receptor Wait Proposal
    a. Minor wording amendments were offered, not changing syntax or semantics.
    b. FV-2, approved 4/0/0.

  5. Container Operations Proposal
    a. Did not cast FV-1 last time because write-up was truncated.
    b. Fixes offered for C/C++ bindings.
    c. FV-1, approved 4/0/0.

  6. Object is Committed Proposal
    a. Small clarification made to binding using MPIRT_BOOLEAN
    b. Genesis of Erratum 37.
    c. Genesis of requirement to check whole document for analogous errors to
    Erratum 37.
    d. FV-1, approved 5/0/0

  7. Int64 Proposal
    a. Editorial corrections made; nothing major.
    b. This proposal is ready to be integrated into the document now.

  8. Container Iterators Proposal
    a. Minor editorial changes, mainly to bindings were offered.
    b. FV-2, approved 5/0/0.

  9. Buffer Iterator Retrieve Proposal
    a. Rename functions to avoid abbreviations
    b. Discussion regarding thread safety, purported utility in practical
    c. Function name changed to MPIRT_BUFITER_SNAPSHOT.
    d. FV-1, approved 3/1/1.

  10. Clique All-to-all Collective Channel Proposal
    a. Modified some text.
    b. Satisfied ourselves that bindings are correct; no syntactic or
    semantic changes.
    c. FV-1, approved 5/0/0.

  11. Errata to MPI/RT 1.0 Standard
    a. In addition to posting with MPI/RT 1.0, post with MPI/RT 1.1 document.
    Certain points suspended pending 1.1 final outcomes, and will be omitted
    from 1.0 posting.
    b. FV-1 (including #37 as slightly amended), 5/0/0

  12. Strided Buffer Proposal
    a. Generalizing the capability cf., Figure 5.7, also pages 100-101.
    b. MPIRT_BUFFER_CREATE_SUBBUFFER_STRIDED(inbuffer, byte_displacement,
    element_offset, blocksize, byte-stride, nblocks, outbuffername,
    c. Significant discussion ensued.
    d. Updated proposal will be prepared by Tony/Cui.
    e. Straw vote, approved 5/0/1.

  13. Variable Buffers Proposal
    a. Proposal + fixes + actions were defined on meeting copy.
    b. Updated proposal will be prepared by Tony/Cui.
    c. FV-1, approved 4/0/1.

  14. Arrangements have been made to hold the next MPI/RT Forum and Data
    Reorganization Forum meetings in San Diego on the following schedule:

    Wednesday, February 28th, 0800-1700 – MPI/RT
    Thursday, March 1st, 0800-1700 – Data Reorg
    Friday, March 2nd, 0800-1300 (incl. lunch) – Data Reorg

    The meetings will be held at:

    Best Western Island Palms Hotel and Marina
    2051 Shelter Island Drive
    619-222-0561 or 800-922-2336

    There will be a meeting fee of $50 per person per day. If you plan to
    pay by check, please make it payable to “SSC-SD Conference Fund”.

    Please let Dennis Cottel ( know by the morning of
    February 22nd if you will attend so the hotel can be given a firm count.

Minutes for the MPI/RT Fourm meeting at MITRE, Boston on 18 September, 2023

Murali Beddhu MPI Software Technology, Inc.
Ken Cain MITRE
Dennis Cottel SPAWAR
Zhenqian Cui MPI Software Technology, Inc.
Yogi Dandass Mississippi State University
Nathan Doss LM-NESS
Arkady Kanevsky Mercury
James Lebak MIT Lincoln Laboratory
Jothi Padmanabhan Mississippi State University
Anna Rounbehler
Tony Skjellum MPI Software Technology, Inc.

  1. There was a lengthy discussion regarding the future of the MPI/RT forum.
    Arkady Kanevsky offered a motion to “mothball” the standard and stop
    meeting. Since there was nobody willing to second this motion, an
    alternate proposal was offered by Dennis Cottel, seconded by James Lebak.
    This motion proposed to:
    a. complete the MPI/RT 1.1 document by June 2024,
    b. reevaluate the direction of the forum, and additional activities that
    may be undertaken eg. Fault Handling.
    c. emphasize the completion of the MPI/RT 1.1 standard as opposed to the
    totality of proposals resolved.
    Proposal accepted by formal vote: 6/1/1

  2. List of proposals with a relative weight of remaining effort required
    Proposal Owner Vote Effort to complete

Errors Dennis C,C 1
Error string Dennis C 0
Sleep Dennis F,C 0
Receptor wait Dennis F,C 0
Bufiter buff return Cui C 0
Int64Export Arkady F 0
Container Ops Dennis C,C,C 0
Container Iter Nathan C,C,C 0
All-to-all collective Arkady C 2 new owner: Yogi
All-to-all bipartite Arkady C 7 new owner: Yogi
Strided buffer Arkady C 7 new owner: Cui
Variable length buff Steve C 8 new owner: Cui

vote: C = Continue work to improve proposal
F = Formal vote to accept

  1. Change hierarchy figure to reflect container iterator changes

  2. Error String proposal:
    a. Fix bindings
    b. The result string is a constant string; users cannot modify or
    free the result string.
    c. Formal vote: 6/0/0.

  3. Sleep proposal
    a. Fix C++ binding
    b. 2nd Formal vote: 6/0/0

  4. Error proposal
    a. Formal vote: 5/0/1

  5. Receptor wait proposal
    a. Fix C++ bindings
    b. Sections 10.2 and 10.3, change “main program” to “program”
    c. Move section 10.2 to section 10.2.2 and section 10.3 to
    section 10.3.2.
    d. 2nd Formal vote: 5/0/1

  6. Container operations proposal
    a. Add a proposal for container wait (all, any, some)
    b. Remove search order parameter and remove error invalid mode
    proposal accepted by straw vote: 5/1/1

  7. MPIT_INT64_EXPORT32 proposal
    a. Fix C++ bindings (MPIRT::INT64_EXPORT32)
    b. 2nd Formal vote 6/0/0

  8. Container Iterator proposal
    a. 1st formal vote 5/0/1

Minutes for the June 6-7, 2023 MPI/RT meeting hosted by Mercury Computer Systems, Inc. in Boston

Dimitris Christodoulou SKY
Dennis Cottel SSC-SD
Zhenqian Cui MSTI
Yogi Dandass MSU
Nathan Doss LMCO
Arkady Kanevsky Mercury
James Lebak MIT/Lincoln Lab
Tom Murphy UMAS Lowell
Steve Paavola SKY
Michael Pepe Mercury
Anna Rounbehler
Jeffrey Smith Mercury

  1. There was a discussion on how we can make MPI/RT more popular. There was general consensus that a public domain version was needed to demonstrate the API. However,
    no funding sources for this implementation exist as yet.

  2. Dennis Cottel gave a status review of the Test Suite.
    a. The test suite has been executed using the Mercury and CSPI implementations
    b. Dennis is waiting for access to the SKY computer at Rome Lab
    c. Dennis will put a copy of the test suite on the MPI/RT web site so others
    may exercise the test suite independently.
    d. SKY Computers will use the publicly released test suite on their platform

  3. Errors Proposal
    a. Arkady will post rephrased wording for paragraph on lines 23-25 on page 15.
    b. Function arguments are checked in a strictly left-to-right manner based on the
    LIS definition. This needs to be specifically stated for C implementations.
    In C++, the object whose member function is being called is checked first.
    c. Create a chapter/appendix to keep deprecated objects and constants. For example,
    d. The list of errors will continue to grow as the test suite uncovers additional
    error conditions.

  4. Error string proposal
    a. The function should return a pointer to an error string.
    b. The production mode library can return any valid null terminated string (i.e.,
    does not have to return a pointer to an error string, an empty null-terminated
    string is permitted).
    c. This proposal should be incorporated in the document as section 1.4.4
    d. Straw vote to continue work on proposal: 8/1/0

  5. Errata straw vote: 6/0/0

  6. Sleep proposal
    a. Remove references to “thread” since a threaded model in not required by MPI/RT.
    b. Replace the reference to MPIRT_TIME_INFINITE with MPIRT_TIMESPEC_IGNORE.
    c. Remove MPIRT_TIME_INFINITE from the end-of-chapter summaries paragraph.
    d. Straw vote to continue work on proposal: 6/0/0

  7. Receptor wait
    a. Minor wording changes.
    b. Straw vote to continue work on proposal: 6/0/0

  8. Bufiter get status proposal
    b. Clarify what the function returns when this function is called before commit,
    after uncommit, and during the real-time (committed) mode of the MPI/RT
    c. The last parameter (vcontainer) of the function is an INOUT parameter.
    d. The size of the vcontainer is not increased to the bufiter size. If the number
    of buffers in the bufiter is greater than the capacity of the vcontainer, the
    function returns error MPIRT_ERR_INV_RANGE
    e. Straw vote to continue work on proposal: 6/0/0

  9. MPIRT_INT64_EXPORT32 function proposal
    a. The C++ binding has an error. The first parameter should have type const
    MPIRT::Int64, not MPIRT_Int64.
    b. Remove advice to users.
    c. Formal vote: 5/0/0

  10. Container operations proposal
    a. The container parameter for all operations in the proposal should be IN only (not
    b. The container must only contain objects that can be started. If the container
    parameter is not a valid container the function returns MPIRT_ERR_INV_CONTAINER.
    All other error returns are those raised by the channel activation function.
    c. The flatten function removes duplicates without raising errors.
    d. The flatten function takes a parameter of type MPIRT_TRAVERSAL_ORDER that
    specifies the traversal order of nested input containers. The values of this
    e. The traversal order determines the relative position of the objects in the resulting
    f. MPIRT_CONTAINER_FLATTEN becomes MPIRT_name-of-container-class_FLATTEN.
    g. The container operations, except for filter, are not applicable to Groups
    h. Remove the MPIRT_ERR_DUPLICATE_MEMBER error. Make this condition the subject
    of an “advise to users” paragraph.
    i. Put a reference to the activate and deactivate functions in the container chapter
    with the container operations.
    j. Straw vote to continue work on proposal: 7/0/0

  11. Container iterator proposal
    a. Change the wording of the paragraph that states that iterators are derived from
    MPIRT_OBJECT to be consistent with the other objects in the document.
    b. Correct the C/C++ bindings for all functions.
    c. Put the GET operations before the SET operations.
    d. In the SET_CONTAINER operation, replace the last two sentences to “This operation
    resets the container iterator.”
    e. Clarify the meaning of “invalidate.”
    f. The last paragraph before the code example, remove the text "(e.g., during a
    real-time phase of processing).
    g. List the meta-classes (list on page 21) over which container iterators are defined.
    h. Add container interators to the object hierarchy diagram (under probe).
    i. Consider adding a small introduction to container iterators in Chapter 1.
    j. Deprecate old retrieve-next container operations (add to the Deprecated appendix).
    k. Straw vote to continue work on proposal: 7/0/0

  12. All-to-all collective (non-bipartite) channel proposal
    a. It was decided to require that the buffers be evenly divisible by the number
    of participating processes.
    b. An error is returned (MPIRT_ERR_ILLEGAL_BUFITER) when the channel is
    created and also when the channel is committed if the buffer size is not divisible
    by the cardinality of the group.
    c. Add a paragraph after the channel creation binding stating that parameter
    consistency checks are performed at channel creation and commit time.
    d. Straw vote to continue work on proposal: 7/0/0

  13. Bipartitie all-to-all channel proposal
    a. Add “bipartite” to glossary
    b. The sentence “The order of dividing a send message into smaller messages…” needs
    to be clarified.
    c. A diagram explaining the data transfer is needed.
    d. The two participating groups must be disjoint.
    e. Add the function bindings.
    f. The condition naming convention for these channels is:
    g. Straw vote to continue work on proposal: 5/0/2

  14. Strided buffer proposal
    a. Reword rationale to clarify and correct.
    b. Need a diagram to describe how strided buffers can be subdivided. There are
    several cases. Provide examples.
    c. Ensure that strided buffers work for all channel types (e.g., reduce, scatter,
    d. No existing calls will change, the proposal extends the buffer object. It does
    not introduce a new object.
    e. Straw vote to continue work on proposal: 5/1/2

Notes from the March 9-10 MPI/RT meeting in Atlanta.

Reviewed the updated document. DRAFT is removed from the cover.
The technical editor recommended changes (e.g., to put in common voice)
but all changes were actually made by MPI/RT editors (Yogi).
The standard, having already been voted on, is complete.

We discussed priorities for 1.1 with several folks expressing the desire
that, given the current lack of experience with 1.0, we should stick to
straightforward subjects that can be handled within a year or so.

The prioritized list:
0. errata
0.1 wait/sleep proposals which have already been voted in except
for exact wording

  1. errors
    1.5 an as yet unwritten proposal to get the two 32-bit parts from
    a 64-bit time
  2. relaxed buffer sizes and offsets
  3. container (a) iterators and (b) operations
  4. additional collective operations

Not making the recommended list for 1.1:
half channels
K-slack channels
generalized mode changes
data reorg capabilities

Discussion of pattern (nee helper) functions: Whether they will be part
of the standard or separate. Since we didn’t have minutes from the
December meeting (why not!) this discussion was (in my opinion) not

Discussion of error proposal: This included how to incorporate it
in the document: a table at the end of the chapter or do all the error
returns need to be discussed in the function text. Three returns were
a function is called before the library is initialized)
the old name will not change but become deprecated
Action items:

Discussion of variable buffers proposal: There were some minor changes and
one significant one: when receiving a buffer of variable count or offset,
both the actual count and actual offset will be set by the system. This
allows the reuse of a buffer without manually resetting its offset.

Discussion of the container iterators proposal: In addition to some wording
changes, and the note that the GET/SET_CONTAINER functions are missing,
it was decided that duplicating an iterator also duplicates the current
position so that the two iterators are indistinguishable after the DUP.
This is so that a copy of an iterator can be made at the current point in
the iteration and returned to later.
Action items:

This led to a discussion of whether a DUP of an object gets you a copy
of the parameters at that moment, or the parameters as they were when
the DUPed object was originally created. Several felt the document said
the latter, while the consensus was that the former was intended – or
at least wanted now. Changes will be made to clarify this and included
in the errata.
Action item:

Another errata item noted: all figures (and tables?) should have numbers,
and there should be a Table of Figures.
Action item:

Discussion of container operations: The difference between regular and
atomic version of all were revisited. The result was that the proposal
would be revised as follows:
there will be a single version of the call for each (remove the atomic
the order of operation is determined by whether the container is a
vector or set: if vector, user specifies; if set, implementation specs.
activate, raise, and start: order means getting the buffer from the
bufiter and enqueing; QOS determines subsequent execution
deactivate: order means wait for each to finish in order (note that
deactivate is essentially a blocking call)
Action item:

Discussion of additional collective channels: Only the important cases should
be addresses to avoid delving into Data Reorg issues which will be a separate
proposal one of these days. The most important cases are
all to all clique
all to all bipartite
exclusive scatter (sender doesn’t receive any)
exclusive gather (receiver doesn’t send any)
The last two are special cases of the second. In the interest of minimal
additions and short time, it was decided to do the first two cases with some
restrictions: the two groups must be disjoint, and (like current gather and
scatter) the parts have to be evenly divisible by N processors.
Action item:

New proposal wanted to retrieve the current buffer count and list from a
buffer iterator. [Action: Cui]

MPI/RT Meeting, September 24, 2023, held at Mercury Computers

The following attendees got down to business at about 9am:

Dennis Cottel, SSC-SD,
Dimitris Christodoulou, SKY,
Nathan Doss, LM/GES,
Mike Melandez, Mercury,
Arkady Kanevsky, Mercury,
Anthony Skjellum, MPI Softtech,

The purpose of the meeting was to review changes made to the document at
the meeting two weeks ago, Sep 8-10, held at SKY. The document reviewed
is dated September 24, 2023 as it was printed in the wee hours on the
day of the meeting.

Several items in the Glossary were revised, as was Figure 1.1, to make
them consistent. The revised Glossary was accepted by a formal vote of

A description of the “name_of_class” notation was added to 1.1.5 and 2.1.

We reviewed essentially all of Chapter 1 since there had been numerous
changes, and accepted it by a formal vote of 4/0/0.

In Chapter 2, for consistency, MPIRT_KEYVAL_DELETE_FN was renamed to
MPIRT_KEYVAL_FREE_FN with all the attendant changes.

Also in Chapter 2, there was a lengthy and arcane discussion of the C++
API with the result that “constructors” are public, not private as
currently shown in the example in 2.7. The minutes from the last
meeting asked that a new section on C++ be added and this had not been
done (as this continuing discussion pointed out). The new section will
be added.

We reviewed primarily the marked changes on Chapters 3 through 15 with
only minor edits and cleanup, including picking up some previously
mandated changes that hadn’t made it into the document.

We adjourned at 2pm.

Minutes from the June 8-10 2023 MPI/RT Forum meeting that was held at LM-GES in Moorestown, NJ.


Arkady Kanevsky (co-chair) Mercury Computers
Dennis Cottel SPAWAR (SYSCEN, San Diego)
Randall Judd SPAWAR (SYSCEN, San Diego)
Nathan Doss LM/GES
Tom McClean Lockheed Martin (LM/GES)
Shane Hebert MSTI
Michael Grieco ASEC/NSA
Steve Paavola SKY Computers
Anthony Skjellum (co-chair) MSTI
James Lebak (minutes taker) MIT/LL
Yogi Dandass MSU

There was 1 official vote on a clarification to MPI/RT-1.0 document.

A. MPIRT_INIT is a required synchronization point for all processes. Formal vote passed 7/0/2.

There were several straw votes on the MPI/RT-1.0 document.

  1. The name for the states should include the name of the object. This means that instead of
    “ACTIVATED” it should be “MPIRT_CHANNEL_ACTIVATED”. See issue 9. Vote passed 7/0/2.
  2. The changes for issue 21. NULL objects for all types. New operation to find out if an object is NULL
    of any type. Dup of NULL object refers to the same object. Vote 7/2/2.
  3. The tables of all possible return codes for each function should not be done for MPI/RT-1.0 document,
    but should be done in MPI/RT-1.1 document. Vote 8/0/0.
  4. No clarification is needed for best effort quality of service for triggers. Vote 9/0/2.
  5. Channel activation is not allowed if in-bufiters for all channel endpoints are shared with other
    channels. Vote 6/0/5.
  6. The changes for issue 27. Clarification for Commit for Best Effort QoS specification. Vote 8/0/3.

There were several formal votes on MPI/RT-1.1 proposals.

  1. The Sleep proposal with changes passes the official vote 7/0/1.
  2. The Receptor Wait proposal with changes passed the official vote 8/0/0.

I. The meeting came to order at 8:30am on Tuesday June 7.

II. The Minutes of the April meeting where approved 8/0/1.
The names of three proposals for relaxing data descriptor requirements will be added to the
minutes. Change Morristown to Moorestown NJ.

III. During most of the meeting the changes to MPI/RT-1.0 document were discussed.
The following list of 46 items for change requests accumulated since the March meeting was presented.
Majority of the items is done already. The remaining items and the changes agreed at this meeting will be
done prior to the next one.

  1. Go through entire document and use \state & \cond macro for the states and conditions of all objects.

  2. In introduction chapter (chapter 1) write sections 1.3.4 (Erroneous programs) and sections 1.3.5
    (Implementation libraries).

  3. Remove “draft” from the cover page (last one to do!)
    Remains open, the last one to be done

  4. C++ binding changes
    Done, see issue 19 for more changes for C++ binding approved by the meeting

  5. Check C bindings (especially for Containers)
    Open. Add in object chapter that for MPIRT_CONTAINER users need to do typecasting to the specific
    container time (CSET, CVECTOR, GROUP) for C binding. No typecasting needed for C++.

  6. Check how we use “transmittable” in glossary, in intro, in dataspec chapters.

  7. Add to the preface “how to read the document” (what advises mean, naming conventions, and so on)

  8. Resize the figures
    Done. Figure 2.1 on page 19 is too large. Reword citations after figures 8.1, 8.2.

  9. Decide if we should add the name of the object before the state (another words instead of
    ACTIVATED it will be MPIRT_CHANNEL_ACTIVATED)? If decision is positive then do global
    change for all state names.
    The straw vote passed 7/0/2 for longer names. The current document (June 7 '99 version on web)
    already reflects this change.

  10. Add examples to the final document.
    Everybody agreed that we need examples. There was a discussion on whether we should add them to
    the MPI/RT-1.0 document, MPI/RT-1.1 document, put them on the mpirt web page or in a separate
    document. It was pointed out that examples are not part of the document and are not subject to the
    vote. At the end it was agreed that the examples will not be a part of MPI/RT-1.0 document but will be
    a part of MPI/RT-1.1 document and will also be placed on the web. All the vendors should check if
    they can “donate” examples. All examples placed on the web and in the document should be tested on
    the vendor’s implemented MPI/RT. MPI/RT editors will try to ensure that they also run on all existing
    MPI/RT implementations (Mercury, Sky, & CSPI).

  11. Define MPIRT_ERR_OBJECT_NOT_COMMITTED error which will be return by a commitable
    object for operations that require the object to be committed. Add this error for all appropriate
    Error is defined in the latest draft. Error description needs some rewording.

  12. For each function define a table with all possible status return values and under what conditions each
    value is return. (May be just add these tables at the end of each chapter rather then in each function
    description) Synchronize with Annex A and the first (status returns) index.
    It was agreed that this should not be done in 1.0 but should be done in MPI/RT-1.1 document. The
    straw vote was taken and passed 8/0/0.

  13. Define a generic error which will be return in all cases where a specific error name is not defined.
    Document all these places and generate a list of suitable error names for all these cases.
    It was decided not to add a generic error. The existing error MPIRT_ERR_INV_OBJECT will be
    changed to MPIRT_ERR_INV_ARGUMENT to handle generic invalid parameter to MPIRT operation.

  14. Clarify what data is in the out-bufiter if the transfer is not successful. Clarify how an implementation is
    allowed to abort the transfer.
    Data in the “last” buffer is undefined if the error is returned. The buffer is still moved from input to
    output buffer iterator in the case of error. If QoS is pure priority or best effort the
    MPIRT_ERR_TRANSMISSION is returned. The related condition is defined for the channel object so
    that users can define a receptor and handlers to deal with it. For all other QoS specifications
    (deadline, timeout or stop event) the existing QoS error should be returned. For MPI/RT-1.1
    investigate what can be done with transmission error status. Remove “Abort” (for channel
    transmission) from the document.

  15. Can objects (commitable) be decorated at any time? After Commit?
    Yes, objects can be decorated at any time. Add a note to the text on that.

  16. Elaborate on scatter_channel (as it is done for gather_channel).
    Done. Add a sentence that the receive buffers for all channel endpoints are the same. Add analogous
    sentence for gather channel also. Add the figures for the definitions for collective channels from the
    MPIDC MPIRT tutorial.

  17. Define a missing CVECTOR_NULL in text.

  18. All operations should return int. Fix returns for operations in 3.2.3.

  19. Dup should be defined for all object not just leaves of the object hierarchy (implications for bindings)
    The discussion of this issue quickly centered what to do with operations that take non-leaf parameters.
    It was agreed that the current DUP definitions are wrong. The current DUP allows both copy
    constructor and clone. For leaf classes DUP operation will be define explicitly and no type casting is
    needed. Add advice to the users that for some operations on virtual classes type casting may be needed
    for some language bindings. For functions that takes non leaf type parameters (CSET_INSERT,
    CVECTOR_REPLACE, DUP, FREE) type casting is needed. Copy constructor should be described
    separately from DUP. Finally the following solution was agreed upon:
    a. Default constructor which initializing to object_NULL (of appropriate type). [int =
    obj1.create(.) ? user. int MPIRT::object::create(.);]
    b. Create fills it (C++ returns error code just like C)
    c. Dup is not copy constructor. (C++ returns error code just like C) [int=obj1.dup(obj2) ? user
    use; int MPIRT::obj1::DUP(obj2) ? C++ binding] Dup is defined for MPIRT_OBJECT. All
    other objects inherit it. Type casting is needed even for leaf. Users can either call dup on leaf
    object type directly or use dup of virtual class with type casting. No ==.
    d. Explicit free call separate from destructor. Use the same explanation as for constructors.
    [Rationale: No exception calling, error returns code instead. Exception handling too expensive.]
    For C++ add Free before destructor. Add a separate section for C++ for constructors and

At the second day it was pointed out that C++ provides copy-constructor and op = automatically. That
means that an MPI/RT object is handle to the actual object and that = and copy-constructor do byte-
by-byte copy of an object handle and the new object is IS_EQUAL to the original object as defined by

  1. Define MPIRT_OBJECT_NULL. (static member of base class)

  2. What are the NULL objects for non-leaf classes? Do we have them? Do we need them?
    Yes, we need NULL for all objects. Add new functions MPIRT_OBJECT_IS_NULL(obj,flag) which
    returns that the flag is MPIRT_TRUE if the object is a NULL object of any type.
    MPIRT_<object_class>_NULL constant has object type. NULL objects have object name of empty
    string “”. Duplication of NULL object of any type refers to the same NULL object. This means that
    MPIRT_OBJECT_IS_EQUAL for a copy of the NULL object and NULL object will return
    MPIRT_IS_EQUAL. The straw vote was taken for these proposed changes, which passed 7/2/2.

  3. What should is_equal operation return? (binding implications). What does it return for “comparison”
    for object_null, channel_null, pt_channel_null?
    See previous issue.

  4. What is the meaning of the group WORLD if INIT is not a synchronization point?
    MPIRT_INIT is now a required synchronization point for all processes. Formal vote passed 7/0/2.

  5. Should we add an error MPIRT_ERR_TIMEOUT for commit to return when one of the group
    WORLD members was not created?
    It was a general consensus that timeout for commit is useful and desirable. However it was decided to
    postpone this until MPI/RT-1.1 where is part of the QoS of commit issue and is a part of the RT mode
    change proposal.

  6. Clarify which channel will transfer a buffer for shared bufiter for BEST_EFFORT QOS.
    This discussion was spread over two days. There are two real issues: one is what guarantees does
    implementation provides for best effort (and priorities), and which channel should transfer a buffers in
    the case of shared bufiters.
    For the first question the agreement was reached that if the resources are available then the progress
    shall be made according to the quality of service. This means that for the case of priorities, if
    resources are available the highest priority channel will transfer among all channels that are ready to
    transfer a message. In the case of equal priority or best effort (the lowest priority) any channel can
    make progress if resources are available and no higher priority channels are ready and if no
    scheduled messages exist for time-driven QoS spec. Replace “infinity” with bounded deadline (can be
    very large) in the definition of best effort on page 165. Clarify that if resources are not available then
    no progress will be made for a channel message transfer for best effort and priority QoS
    specifications. The lower priority channel can be delayed indefinitely if resources are not available.
    Remove advice to the users on page 165.
    For the second question there were several sub-issues to clarify. First, the clarification will be made to
    description to the START operation on page 149 to unsure that the buffer is removed from the input
    bufiter when the start returns. The completion of start is instantaneous and does not mean that the
    transfer had started. The situation with ACTIVATED is more complicated. The small state transition
    diagram for activated case was accepted. There are three states inside current activated state: Ready,
    Xferable, and Xfer data. When the buffer is inserted in an activated channel in-bufiter the channel
    transitions into Cferable. If some other channel (or user) removes the buffer from the bufiter and the
    input bufiter becomes empty the channel transitions back into Ready state. From Xferable when all
    other channel endpoints are in Xferable the channel endpoints all transition into Xfer data and
    transfer message(s) over the channel. At the completion of the transfer the channel transitions into
    Ready state again. This clarification will be imbedded into the figure 9.1 on page 153.
    The table was created which will be added to the document that defines what happens for channel
    endpoints in the case of started/activated and in the case of shared and not shared bufiters. All the
    cases are already defined except the case of shared bufiters and activated channel endpoints. For the
    cases of activated channel endpoints if only one side: send or receive has shared in-bufiter this is
    equivalent to get and put models respectively. The case when channel endpoints are activated and their
    in-bufiters are shared is NOT allowed for MPI/RT-1.0. This ensures that there are no deadlocks. The
    last statement had a straw vote that passed 6/0/5.

  7. Clarify the meaning of MPIRT_QOS_BEST_EFFORT for triggers.
    No special clarification needed. Straw vote 9/0/2.

  8. What is the requirement for MPIRT_QOS_BEST_EFFORT from COMMIT?
    Commit succeeds even though there is no “bandwidth” for that best-effort channel?
    There is a time at the end of a very long period?
    No need for changes. Issue 25 already covers it. Straw vote 8/0/3 passed.

  9. Clarify the requirements for a user for putting/getting buffers to/from bufiters for barrier channel. Can
    the bufiter_null be used for that type of channel?

  10. Check Anna’s email comments that she sent to us.
    a. Anna’s glossary comments
    b. Anna’s comments on commit and data transfer
    c. Anna’ detailed list of comments


  12. Agree on the format of for C++ for QOS and DATASPECS (should be a part of 4).
    Done. There are various consistency problems with indexes. Define in footnote what bold page means

  13. Add underlying assumptions of reliability and ordering of messages (appropriate to QoSs)

  14. In 2.2.5 remove that group is “read-only”. Remove all “read-only” in other places.
    Done. Add that GROUP_WORLD, GROUP_SELF and GROUP_EMPTY are all constants and are
    read-only. All constants are read-only.

  15. Add definition of reliability to Glossary.

  16. Clarification for MPIRT_CONTINUE return for the last synch handler

  17. Fix …COND_QOS_FAILURE and …COND_FAILURE as well
    as analogous cond for triggers and receptors. Check against analogous
    cond for channels.

  18. Clarify what is rembuffer for CVECTOR_CREATE. A “semiformal”
    votes by email agreed on MPIRT_BUFFER_NULL.

  19. Clarify that recoverable and retryable operation are only for
    standard defined operations and not user-defined ones.

  20. Fix MPIRT_Int64_assign32 from int to unsigned int.

  21. Clarification and advice to users how to create and manipulate groups.
    Done. More editorial changes needed. Remove the second half of the first paragraph of 2.5.2 (group
    manipulation are defined - inherited from vector container). Change rationale to regular text. Define
    “hole” as NULL object. Add a note that instantaneously a group can have duplicates but none at
    commit time.

  22. Clarify the effect of changes (including free) of the original buffer on buffers that are derive from it
    using buffer partitioning operations.
    The original “big” buffer must be committed with all its children. All the buffers over the same
    SYSTEM_MEM_ALLOC must be committed together (user requirement). Free of buffers over the same
    memory must be done in the opposite order. Clarify that system allocated memory is allocated at
    commit time and free at uncommit time (and commit if a buffer not present). The characteristics of the
    original buffer (first to specify shared memory) cannot be changed if there are buffers derived from it
    (either dup without base change or buffer partitioning). The memory linkage is always to the first
    buffer that is created over the same memory using SYSTEM_MEM_ALLOC. Dereferencing (changing
    base address) of any of the intermediate buffers over the shared memory has no effect on its children
    who are still referencing the original buffer memory. Add an error if the system alignment for the
    dataspec is broken by the change of dataspec or base address.

  23. Add definition of reduce operation (from MPI-1).

  24. Clarify receptor duplication operation and event name in particular.
    Done. Copy the new text from section 4.2.2 (common EVDEL operations) to section 4.2.4 (receptors).
    Other small editorial changes.

  25. Fix event name problems
    There is a problem with uniqueness of event names. The event delivery abstraction scope for MPI/RT-
    1.0 is GROUP_WORLD. That means that the event name must for be unique over GROUP_WORLD.
    The forum considered a proposal of adding a group to the event name for implicitly generated events.
    For explicit events it is a user responsibility to ensure the uniqueness of the name. The advice to the
    user should be added to clarify this. As part of this discussion the decision was made to add a
    requirement that the name of the group object must be the same for all members (and users- see
    discussion below) of the group (processes of the group). This will simplify an implementation job of
    matching all the processes of the group to ensure that the group definition is correct. The old
    requirements of the same task address in the same order in the group (vector container) remains. A
    new error MPIRT_ERR_GROUP_INCONSISTENT will be added. This error covers both duplicate
    group names and that not all group processes created the right group.

    The old specification for the channel object implicitly generated events is a STRING_NAME of type
    “channel name”:rank1:rank2:“condition name”. This requires that the channel name is unique over
    the GROUP_WOLRD. The same channel name must also be unique over the group over which the
    channel is created. There was a discussion to change the format to group:rank:object type:object
    name: condition for all implicitly generated events. After long discussion it was agreed that it is not
    needed for MPI/RT-1.0 fix.
    All objects that can generate implicit events except channels are generate within one process and
    should have a unique name within this process. The rank is part of the event name for them so we can
    think of them as scoped over the group GROUP_WOLRD. Channels are scoped over the group over
    which they are defined. It is user responsibility to ensure that if there are receptors for the channel
    events then the channel name must also be unique over group world and hence unique in the process
    where endpoint is defined. Thus, channel event names become analogous to the other object event
    name if “rank 2” is dropped. The receptor’s process for channel events does not have be a member of
    the group over which the channel is defined.
    The following two resolutions were made: “rank 2” is not needed for channel event names. When
    defining an implicit receptor, the scope passed to the receptor needs to be the same as the group over
    which the object is defined, and the receptor process does not have to be a member of that group. This
    means that if a user would like to have an implicit or an explicit receptor for an implicit event, the user
    needs to create a group over which the object generating the implicit event is defined (group of a
    channel and MPIRT_GROUP_WORLD for the rest) and the user needs to pass that group as an input
    parameter to an explicit receptor or a QoS specification for implicit receptor.

  26. Define object names for object created by implementation (e.g. GROUP_WOLRD, GROUP_SELF)

  27. Clarify that buffers in the initial buffer list are inserted into the buffer iterator at commit time.

V. At the next meeting the full review of the MPI/RT-1.0 document will take place with the formal

VI. The next Forum meeting will take place at Sky Computers on Sept. 8-10 W-F (half day Friday)
(no Data Reorg). Data Reorg will take place at HPEC.
The meeting after that will take place at Mercury Computers on Oct 19-22 Tu-Fr. MPIRT meeting
will be 2 days Tu-Wed. Data Reorg 1and a half Th & Fr. Friday is half day.
The final meeting of the year will be in San Diego Dec. 14-17 Tu-Fr. MPI/RT- 2days, Data Reorg
1 and a half. Friday is half day.

VII. The k-slack proposal was discussed.
There were several requests to clarify things. First, what is the relationship between QoS and k-
slack? Second, the current proposal draft is geared toward 0-sided communication. Please,
defined what happens for 1- & 2-sided models. Are multiple starts allowed with k-slack? What is
the definition of WAIT for the k-slack channel? The same question for TEST, and channel
conditions and implicit events. Add functionality so that a user can “wait” for a buffer associated
with a particular message transfer (transaction).
As part of this discussion the Forum asked for a clarification for MPIRT_CHANNEL_WAIT in
MPI/RT-1.0 document on page 151 lines 13-27.

VIII. Container Iterators proposal was discussed.
The container argument should be flat (no nested containers of any type) and contains only
objects of the type on which the operations can be performed. Two new functions FLATTEN aand
FILTER should be added to help with that. The order in which the operation should be performed
on the objects in the container is undefined and implementation is free to apply the operation on
objects in parallel. New errors will be introduced if the input container is not flat or contains
objects of the wrong type.

IX. Sleep proposal was discussed.
What happens on mode changes? Sleep should return when Commit or Uncommit is called. The
return code should specify why returned. If specified time is an absolute time which had passed an
error should be returned. The formal vote on the proposal had passed 7/0/1.

X. Receptor wait proposal was discussed.
What happens on mode changes? Receptor_Wait should return when Commit or Uncommit is
called, or when receptor goes away. The return code should specify why returned. The formal vote
on the proposal had passed 8/0/0.

XI. The testing suite for MPI/RT-1.0 was discussed.
The Forum should endorse it. For outreach a new webside “next” to mpirt web site should be
created with cross pointers. What should be done for the first spec? By december it has only time
for a small subset. What functionality should it include? Initially in C without eliminating C++.
Initially should not concentrate on what happens for erroneous programs. Meta standardization:
run command, error formats, hidden flags and access to implementation internals. Debug vs.
production mode libraries.

XII. The proposal for relaxing data descriptor requirements was discussed.
Change “get” to “retrieve”. Change MPIRT_INT to “int”. It should represent the number of
elements of the buffer dataspec. Change “length” to “count”. Offset should be non-negative and
is from the start of the buffer. The current proposal is geared to ptchannel, specify what effect
these changes have for collective channels. Define relationship between buffer count + offset to
the bufiter count. All dataspecs should match. Specify what size buffers can be inserted in bufiter (
for variable length and offset) and which part of the buffer is transferred (send side, receive side).
The count of the received buffer is modified by the channel to reflect the received message size.

XIII. Two new topics where added to MPI/RT-1.1 list of topics:
1. Allow Init, Finalize and new Init.
2. Helper function for freeing all buffers over specified system_mem_alloc.

XIV. There was a small group discussion while Data Reorg was in preparation. There was a discussion
on old Compare operation and object equivalency. Users themselves can decide what equivalency
mean for each object type. Two objects for which it is hard to do are buffers and task_addresses.
The following decisions were made to handle that:
1. For MPI/RT-1.1 we need MPIRT_BUFFER_GET_MASTER (buffer, &master, &offset)
operation which returns the first buffer that requested the shared system_mem_alloc memory.
The offset will be in bytes to handle the situation when dataspecs where changed in the tree of
created buffers over the same memory. Hence, specifying the offset in dataspecs is not
2. For MPI/RT-1.1 add MPIRT_DATASPEC_SIZEOF(dataspec) operations to find the size for
dataspec in bytes.
3. For MPI/RT-1.0 remove DUP and FREE operations for TASK_ADDRESS.

Minutes from the April 7-8 2023 MPI/RT Forum meeting that was held in Bedford, MA.

These minutes were approved with a vote of 7/0/1 by the forum during the June 2023


Arkady Kanevsky (co-chair) Mercury Computers
Charles R. Petersen AF Research Lab, Rome
Ken Cain (host) MITRE
Manoj Apte MSU
Dennis Cottel SPAWAR
Nathan Doss LM/GES
Tom McClean Lockheed Martin (LM/GES)
Anna Rounbehler Self
Henk Spaanenburg Mercury Computers
Shane Hebert MSU
Zhenqian Cui MPI Software Technology Inc.
Raghu Angadi MSTI
Michael Grieco ASEC/NSA
James Lebak MIT/LL
Steve Paavola SKY Computers
Dimitris Christodoulou SKY Computers
Rick Massary NAWCAD
Chris Lirakis BBN Technologies
Matt Daily BBN Technologies
Thomas McMahon MSTI
Srigurunath Chakravarthi MSTI
Anthony Skjellum (co-chair) MSTI

There wers 1 official vote on a clarification to MPI/RT-1.0 document
and several straw votes on the MPI/RT-1.1 proposals.

  1. The meeting came to order at 8:30am on Wednesday February 3.

  2. The clarification was requested on inserting and removal of NULL element
    into CSET.

    The following clarification was officially voted on:
    �Insertion of NULL object of any type into CSET does not do anything (no op)
    and returns MPIRT_SUCCESS without any changes to CSET.
    Removal of NULL object of any type from CSET returns an error

    The official vote was 11/0/0.

  3. The proposal on Container Iterators (version 5) (Nathan) was discussed.
    The three options (two stated in the proposal and one by Greg email)
    of what happen to the container iterator while the container is modified where considered.
    The straw vote was taken to remove options 1 and 3 and leave only version 2.
    The vote passed 13/0/4.
    It was agreed to leave MPIRT_CSET_RETRIEVE_NEXT for backward compitability.
    The whole proposal was straw voted on and passed 14/0/2.
    Nathan agreed to convert the proposal into a chapter which will be merged with MPI/RT-1.0
    document (after all the clarifications will be done) which will be a base for MPI/RT-1.1 document.

  4. The proposal on Receptor Wait (version 3) was presented by Dennis.
    The following changes were agreed to for the proposal:
    A new section 14.5 for �timed waiting� will not be added.
    The �receptor� parameter is no longer optional, so MPIRT_RECEPTOR_NULL can not passed into
    MPIRT_RECEPTOR_WAIT. Separately, There was agreement to create an MPIRT_SLEEP functionality.

The modified proposal passed a straw vote 18/0/3. Dennis agreed to update a proposal and identify all
the changes need for it in MPI/RT-1.0 document. The bulk of the proposal will go into Chapter 4.

  1. The sleep proposal was discussed.  Functionality identical to the 

MPIRT_RECEPTOR_WAIT for the NULL receptor was agreed upon. The MPIRT_SLEEP function
will provide that.

The straw vote on the proposal passed 18/0/1. Dennis agreed to update a proposal and identify all the
changes need for it in MPI/RT-1.0 document. The bulk of the proposal will go into Chapter 14.

  1. The Unreliable channel proposal (version 1) by Arkady/Leonard was briefly discussed.
    It was agreed that a clear definition of what an unreliable channel is is needed
    as well as the requirements for the delivery of a message on it. The meaning of QoS for this type of
    channel also need to be clarified. No vote was taken on this proposal.

  2. The Half channel proposal by Shane was discussed.
    It was agreed that the wait_function is not needed. The stop_function was renamed abort_function.
    The need of xfer_complete for the status of the transfer completion was discussed and will be added
    for the next version of the proposal. The discussion of relationship between a user, MPI/RT and device
    vendor of the �remote� endpoint of half channel was discussed. It was reduced to “who provides the
    half channel functions that are part of the half channel definition.”
    Shane agreed to update the proposal for the next meeting. No votes were taken for this proposal.

  3. The k-slack channel proposal by Shane was dicussed.
    The relationship between this specification and QoS was considered. A clear indication how this
    functionality will benefit the MPIRT users was requested. The need for this functionality for the
    MPI/RT-1.1 vs. MPI/RT-2.0 was also considered. Shane agreed to investigate these issue further for
    the next meeting.

  4. The RT mode changes mode proposal (version 1) by Arkady was discussed.
    It was agreed that we will keep existing COMMIT and UNCOMMIT operations but they will be
    written in terms of the RT mode functionality for a single transaction at a time form.
    Since some object should be passed around from one mode to another without really being used but
    simply for their state preservation a FLAG for identifying these object in the �mode container� should
    be added. No sets operations should be allowed on all objects after admission test (even if they are not
    used in the current RT mode), since these objects promised to be unchanged for future transits.
    Since admission testing is a separate functionality now an Admit for RT mode should be added.

There was a long discussion on a situation when an object is in a certain state on starting RT mode of
the transition but need to be at a different state at the end of the transition for the new RT mode. The
prime example of it is a buffer iterator initial buffers which may be in some other buffer iterator when
transition starts. It was agreed that for this proposal we should state that this situation is possible and
that users can prevent it by having empty initial buffer lists for buffer iterators. Users can insert buffers
into buffer iterator themselves after transition.

It was pointed out that the MPI/RT-1.0 should clearly state that the buffers from the initial buffer list
are required to be in the buffer iterator only after COMMIT.
All parts of the �global� object must specify the same information.
Admission_test is synchronous and collective.
Arkady agreed to update the proposal for the next meeting. No votes were taken on this proposal.

  1. Three proposals for relaxing data descriptor requirements between buffer and buffer iterator and
    across a channel were presented by Steve and Shane. The proposals are as follows:
    a) relax the restriction that buffer dataspec and length must match the bufiter specs,
    b) allow variable length data transfers, and
    c) allow the transfer start address to be offset from the start of the buffer.
    Each proposal was first discussed in isolation. The implication of one proposal on the others and
    possible ,configurations of proposals were discussed. It was agreed that more details are need for that.
    Steve agreed to provide these for the next meeting. No votes were taken on this proposal.

  2. The future meeting were considered. The next meeting was scheduled for June 8-10, Tuesday �
    Thursday, with the last day half day meeting in Moorestown New Jersey. Nathan Doss will host the
    meeting and will provide all the information. (Data Reorg will take place on Thursday second half of
    the day and Friday � June 10-11.)
    The other future meeting will be held in late August, October, and December.

  3. There were many clarifications and corrections requested for the current MPI/RT-1.0 document:
    A. Defined an error that will be returned for commitable objects operations that are only defined for
    when an object is committed in the case when the object is not committed, MPIRT_OBJECT_NOT_COMMITTED.

B. Document which errors each operation can return.

C. Clarify what data is in buffer in the out-bufiter if the transfer not successful.
How is implementation allowed to abort a transfer.

D. Can objects (commitable) be decorated at any time? After Commit?

E. Elaborate on scatter_channel (as it is done for gather_channel).

F. Define a missing CVECTOR_NULL in text.

G. All operations should return int. Fix returns for operations in 3.2.3.

H. Dup should be defined for all object not just leaves of the object hierarchy.
(implications for bindings)

I. Fix C++ bindings. Does it have extra * or not? Check C bindings especially for container

J. Define MPIRT_OBJECT_NULL. (static member of base class)

K. What should is_equal operation return? (binding implications). What does it return for
�comparison� for object_null, channel_null, pt_channel_null?

L. What is the meaning of the group WORLD if INIT is not a synchronization point?
Should we add an error MPIRT_ERR_TIMEOUT for commit to return when one of the group
WORLD members was not created?

M. Clarify which channel will transfer a buffer for shared bufiter for BEST_EFFORT QOS.

N. Clarify the meaning of MPIRT_QOS_BEST_EFFORT for triggers.

O. What is the requirement for MPIRT_QOS_BEST_EFFORT from COMMIT?
Commit succeed even though there is no �bandwidth� for that best-effort channel?
There is a time at the end of a very long period?

P. There are several others that were send electronically to the exploder and not presented at the

The editorial board chaired by Tony and Arkady will clean up MPI/RT-1.0 document and will
Present an update document at the next meeting. The binding issue will be resolved within two weeks.

  1. Mike Greco provided a written update on RT-CORBA and related OMG activities.

Minutes from the Feb. 3-5 2023 MPI/RT Forum meeting that was held in San Diego, CA.


Arkady Kanevsky (chair) MITRE
Manoj Apte MSU
Randall Judd SSC-SD
Dennis Cottel (host) SSC-SD
Leonard Monk CSPI
Nathan Doss LM-GES
Shane Hebert MSU
Darwin Ammala MSTI
Clay Taylor, Jr MSTI
Michael Grieco ASEC/NSA
James Lebak MIT/LL
Steve Paavola SKY
Rick Massary NAWCAD

There were no official or straw votes.
The following action items were discussed and
assigned for the proposal writing:

A. Arkady - RT-Mode changes (topic 6)
B. Shane - K-Slack channels (topic 8)
C. Steve - Data descriptors (topic 11)
D. Dennis - Sleep, wake up, wait on (topic 15)
E. Shane - Bufiter Dataspec homogeneity (topic 22)
F. Arkady & Leonard - Unreliable channels (topic 23)
G. Shane - Half channels (topic 24)
H. Steve - Relocatable buffers (topic 43)
I. Steve - Variable length buffers (topic 44)
J. Leonard & Arkady - Unique ID (topic 46)

H. Arkady - Update channel state transition diagram and write a description for
it for MPI/RT-1.0.

  1. The meeting came to order at 8:30am on Wednesday February 3.

  2. The updated list of action items for the MPI/RT was discussed.
    The following items were added to the list over the three days:
    a. Light-weight handlers
    b. Half channels
    c. Default handlers
    d. Group parameter scope for the commit
    e. Mechanisms for passing in externally build schedules
    to MPI/RT
    f. Logical grouping of event kinds (arithmetic on events)
    g. PGCOs
    h. QoS for handlers (add deadlines)
    i. Mutable buffers (late binding of address)
    j. Variable length buffers
    k. Periodic triggers
    l. Unique Id for running MPI/RT version
    m. QoS for QoS (granularity for raising errors)

  3. The topic of real-time mode changes was discussed (topic 6).
    First, the short summary of the history of the topic
    was presented. The objects MPIRT_RESOURCE and MPIRT_TRANSIT
    were reintroduced. A general discussion on resource allocation,
    resource reservation, admission test, and transition between
    real-time mode took place.

    It was decided that for now we should restrict this proposal
    to MPIRT_GROUP_WORLD and only after that is completed consider other groups.
    This decision was based on the following question:
    How do an implementation deal with transition with QoS if it
    does not know when objects are in use by other groups?

    The agreement was reached that there is at most one active
    real-time mode per process. This ensures that all objects are
    available for transition.

    Trade-offs of creating resource objects without knowing future
    transitions vs. knowing the entire graph of all transitions
    were considered and discussed. It was pointed out that the same
    object may be created by two different resource objects
    differently, that may not allow transition between them
    preserving the state of the object or may not allow transition
    with the requested transition QoS. Some discussion concentrated
    on an “incremental” admission.

    The QoS specification for real-time mode changes was discussed.
    It was agreed that initially we should consider only best effort.
    The deadlines and events that trigger transition should be
    discussed after that. A synchronization of transition from one
    real-time mode to another on different processes (processors, nodes)
    was considered as part of this QoS discussion.

    The topics of the relationship of MPI/RT to the OS, hardware,
    and other parts of the system were also partially discussed.

    We agreed that there is a need to support some objects to remain
    operational during a transition (background channels). The effect
    of this to support layered library was also considered.

    Arkady agreed to write a proposal that allows to build the
    two discussed objects in total isolation, with partial transition
    graph, and with the full transition graph. It is up to the user to decide
    which information to provide for the creation of these objects.
    This proposal will also support operating objects during transition.

  4. A concern was raised to the lack of examples in the document.
    A tutorial will be presented in MPIDC’99 and will be available
    on MPI/RT web after words (mid March). The example chapter will be
    updated and be included in the final document of MPI/RT-1.0.

  5. Half channels (topic 24) was discussed. The following issues
    were brought forward:
    a. periodic half channels,
    b. data streams,
    c. interrupts to pull data to half channel buffer,
    d. aperiodic interrupts,
    e. event kinds for half channels (only receptors), with triggers
    as I/O interrupts,
    f. interface consistency with existing channel end points specification:
    buffers, bufiters, QoS and so on,
    g. admission test and QoS specification for half channels,
    h. devices (MPI/RT does not control devices, drivers are outside of MPI/RT),
    i. data can be “dumped” into a buffer without interrupting CPU,
    j. device registration,
    k. hooking up half channels with devices,
    l. scheduled pulling,
    m. random vs. deterministic half channels (dataspecs, data size)

    A device driver model with defined “operation-func” that will be invoked
    on an interrupt to pull the half channel data into a buffer was accepted.
    A completion notification of half channel message transfer uses the existing
    channel end point functionality of events on a channel and/or bufiter.

    A relationship between MPI/RT and other entities on a platform again
    was considered. A consensus is forming that MPI/RT internally specifies
    constraint on the use of resources (bus, memory, and so on) for other
    entities outside of the MPI/RT.

    An issue of multiple devices “dumping” data into the same half-channel
    was considered.

    An issue of extra states for a half channel was brought forward.

    A tentative template of the half channel create function was written:

    … other standard channel end point parameters …

    The direction of a half channel was briefly considered as well
    as operations on a half channel.

    Shane was “volunteered” to summarize the discussion in the form
    of the proposal.

  6. K-slack (topic 8) was discussed. The main issue it to allow
    an implementation to transfer more than one buffer at a time
    to improve bandwidth. For example, an implementation may be able
    to save on synchronization (“ack”, “nack”). The current MPI/RT-1.0
    specification allows at most one buffer to be in a channel at a time.
    A state transition diagram depicts this requirement. Some protocols
    work more efficiently if multiple buffers can be transferred “together”,
    for example “sliding window” protocol.

    It was agree that “k” means that at most k buffers can be in possession
    of a channel at any one time. But there is no requirement or a guarantee
    that an implementation will take advantage of that feature.

    There was a discussion where we should specify “k”. Early vs. late binding.
    It was agreed that the best place to specify it at the early binding,
    either as the QoS or as a channel parameter. The forum was leaning to the
    later. Channel end points do not require to specify the same slack number.
    At commit time the implementation will negotiate the slack number.
    It should probably return the negotiated number back to the users.

    The biggest discussion centered on new and modified operations on a channel
    to support k-slack. The following four operations are under consideration:
    a. Wait_All - that returns when all “outstanding” transfers complete
    (a “window” transfer is completed),
    b. Wait - returns as soon as the “first” outstanding transfer completes
    (Notice that several of transfers may complete at the same time).
    Other interpretations may also be possible and should be discussed further.
    c. Get_Slack - returns the number of outstanding transfers in the channel,
    d. Wait_on_specific_buffer - returns when the specified buffer is in the
    out-bufiter of the channel. This operation was discussed in great
    details on the last day of the meeting. It may require to “poll” on
    every buffer of the channel and on every channel that may share
    that buffer (or bufiter). Hence, this operation may be quite expensive
    and may adversely effect the performance of other objects. We may consider
    a different operation that will allow user to find out when the n-th transfer
    had completed over the channel.
    Once the proposal is written we will return on the issue of the purpose
    of these operations and what users are trying to achieve with them.
    The relationship between individual Waits and Starts need to be considered
    further as well as the changes needed for the channel state transition diagram.

    Shane again was volunteered to write a proposal as a representative of MSU.

  7. Unreliable channels (topic 23) were discussed. It was clear that the restriction
    that MPI/RT “guarantees that the underlying transmission of messages
    is reliable,” that MPI/RT inherited from MPI, need to be reconsidered.
    [It was pointed out that this corner stone assumption seems to be missing
    from MPI/RT-1.0 document.] After long discussion it became clear that
    the forum does not have a clear picture how this unwritten requirement of
    the “guarantee delivery” is interpreted. For example, can an implementation
    “mask” a member of a group that is not “reliable” so as far as all other
    members of the group “active”? Allowing that together with “unreliable” channels
    will allow other members of a group to “communicate with it unreliably”.
    The dividing line between this topic and fault-tolerance, fault-handling,
    dynamic process management, instrumentation and other topics is blur.

    Consensus seems to be forming that unreliability should be part of the
    channel QoS specification. If a channel is unreliable the requirement
    for the implementation to raise an error when the QoS is missed is removed.
    The implementation can raise that error but users
    should not rely on that for unreliable channels. [This brought a question
    on the QoS error for the best effort QoS for the current specification.
    Also a timing issue when a buffer is moved from in-bufiter to a channel
    and from the channel to the out-bufiter for best effort QoS.]
    [It was pointed out that the definition of reliability need to be added to
    the glossary.]

    The issue of the “global” state view was brought up again.

    Arkady and Leonard agreed to flush out more of these issues and summarized
    the unreliable channel discussion into a proposal.

  8. The data descriptors (topic 11) were discussed. It was pointed out that strided
    DMAs may not improve performance vs. local rearrangement of data in memory and
    then transferring it. This is true for DRAM but not for SRAM. It was decided
    that we will not now consider strides as part of dataspecs but instead let
    users do packing and unpacking themselves. The issue of different dataspecs
    in one buffer was considered as well as contiguous memory buffers vs.
    scatter-gather ones. It was decided not to address these issues for now but
    instead consider two other issues: mutable buffers (late binding of buffer
    address) and variable length buffers. Some of the postponed issues will
    be reconsidered when we address topic 1 of Data Reorganization.

    The definition of structures was considered as part of the above discussion.
    The buffer descriptor consists of three parts: dataspec, size, and address.
    The first two parts of this 3-tuple is the data descriptor.
    The forum agreed to defines a message buffer which is an array of 3-tuples.
    This array along with its size will replace a singleton tuple which is
    currently used for buffer specification. Each tuple describes a contiguous
    memory. A system allocated tuples must be pairwise disjoint for the same
    message buffer, but user allocated tuples can overlap. The data descriptor
    parts of all tuples are still required to be matched between channel end points.

    Steve graciously agreed to write this proposal.

  9. The topic of requiring data descriptors to be the same for channel end points
    (topic 22) was briefly discussed. It was agreed that dataspecs must be matched
    but that we should relax this requirement for the size. [This discussion
    was done independently from the previous one. It was agreed that we will consider
    integration of the proposals only after each one of them is approved
    This flexibility allows users to use the same buffer in multiple channels
    instead of creating multiple buffers over the same memory to be used for each
    channel. This is part of the early binding specification, hence the implementation
    can still set up its channel transfers of the appropriate size during
    commit. Details of allocated size and transmitted size will be flushed out
    in the proposal. This will not effect the buffer specification but allow
    the bufiters for the channel end points to specify different length for
    message transmission.

    Shane will prepare this proposal.

  10. The issue of SLEEP, WAKE-UP, WAIT-ON (topic 15) was reintroduced. The old
    problem of what the “main” application should do while the application is
    running as 0-sided communication with handlers to schedule computation?
    The old idea of IDLE was reintroduced. The need of blocking on something
    was reiterated. The waiting on nothing (IDLE), for time (absolute or
    relative time_spec), and on event were considered. It was agreed that a
    single operation MPIRT_RECEPTOR_WAIT (receptor, time_spec) will allow to
    support all three cases. This operation will be blocking and late binding but the
    “special type of a receptor” will be registered (somehow) at the commit time.
    Time_spec allows to set up a timeout so that operation will wait on both
    time and event. If both specified as NULL then this is equivalent to IDLE.

    Dennis volunteered to write a proposal.

  11. The issue of mutable buffers (topic 43) was considered for the
    first time. [Again the forum decided to consider this is isolation for
    now and not as part of bullets 8, 9 or 12.] It was decided that an operation
    for buffer create will have a flag which indicates that the address is
    mutable at run time. The default value of the flag will be set to
    not mutable to allow backward compatibility. [A more generic version will
    be considered when we will merge proposals for this topic with the next
    one (topic 44).] The effect of user allocated vs. system allocated memory
    as well as bufiter policies that match buffers for channel endpoints will
    be considered as part of this proposal.

    Steve agreed to write this proposal.

  12. The related topic (topic 44) of flexible buffer size was also discussed
    for the first time. It was agreed that we will need two sizes:
    maximum possible buffer size and current buffer size. The interpretation
    of the maximum size for sending vs. receiving end point of the channel
    was discussed. The effect of the push vs. pull model let to the following
    agreement. The buffer size must be greater than
    the message transmission size specified by the bufiter (see bullet 9).
    The backward compatibility need to be addressed as well as the merger
    with previous topic.

    Steve agreed to write this proposal also.

  13. The issue of multiple running versions of MPI/RT was brought up (topic 46).
    The idea of having a “unique” ID for each running version/session was considered.
    The sticky issue is the uniqueness of the ID across reboots.
    The use of time_spec along with platform specific and MPI/RT specific identifiers
    was considered. There was no requirement to get a quick solution. Rather
    a need of to consider this ID for debugging purposes, fault identification,
    and instrumentation for the future. Consensus was froming to treat it initially
    as part of the instrumentation as an extension.

    Leonard and Arkady agreed to work on this proposal.

  14. The issue of the channel state transition diagram was revisited. While
    the desire was to consider the topic of waiting on an activated channel,
    it quickly became apparent that the current figure in MPI/RT-1.0 need some
    work. The meaning of labels on the transition, as well relationship
    of the condition raising to the states and transitions need to be clarified.
    It was agreed that while we will preserve the states that are visible
    to the users we will introduce some internal sub-states to them to help
    to understand the relationship between condition raisings and the
    states. The full description of the figure will be added to the text.
    The figure does not represents the requirements but just a tool to help
    visualize the states, transitions, and possible generate events that a channel
    can do.

    Arkady will update the document ASAP and distribute for further comments.

  15. The next meeting of the forum will take place in MITRE in Bedford (Boston)

    in April 7-8. It will be 2 days (may be a day and a half). Arkady will host it.

  16. The meeting adjourn at 10:30am on Friday Feb. 5.

The combined list of topics with their current schedules is stated below.

            TOPIC                           |  Expected  |            |
    and person(s) in charge                     | Completion | Discussion |

1 Data-reorganization support | M | E |
Tony, James (from DATA REORG EFFORT) | | |
| | |
2 Collective operations (ALL_TO_ALL) | E | E |
Games, Arkady | | |
| | |
3 Dynamic Process Management, | E-M | E |
Parallel Client Server, etc. | | |
Tony | | |
| | |
4 Interoperation of MPI/RT Implementations | L | E |
Tony | | |
| | |
5 Support for Java binding of MPI/RT | L | M |
(other languages too?) | | |
Tony | | |
| | |
6 Full-QOS-based MPI/RT with mode changes | M | 2/3/99 |
(such as QOS capability for mode changes) | | |
Arkady, Steve | | |
| | |
7 Explicit ability to work with real-time | L | M |
schedulers for process scheduling | | |
Manoj | | |
| | |
8 Work to support k-slack channel mode | M | 2/3/99 |
(whereas MPI/RT 1.0 is 1-slack) | | |
Shane, Tony | | |
| | |
9 Support for IDL for MPI/RT | L | L |
Tony | | |
| | |
10 Fine grain, Coarse grain QOS specifications | M | M |
Arkady, Leonard | | |
| | |
11 More powerful data descriptor to allow | M | 2/3/99 |
strided and non contiguous buffers | | |
James, Tony | | |
| | |
12 how to handle other resources’ schedulers. | L | M |
Currently we have dealt with memory and | | |
network in two different ways and via | | |
completely different specs… | | |
(related to 7 and 10) | | |
Arkady, Leonard, Manoj | | |
| | |
13 Operations on containers. The old issues | E | E |
of propagating the operations to every object| | |
in a container (probably for the containers | | |
that contain homogeneous objects only). | | |
Nathan, Dennis | | |
| | |
14 Container Iterators | E | E |
Nathan, Dennis | | |
| | |
Dennis, Arkady, Tony | | |
| | |
16 Channel state transition (old busy state) | E-M | E |
Cui, Arkady | | |
| | |
(related to 16) | | |
Arkady, Cui | | |
| | |
18 QoS Spec for channel (remove restrictions | E-M | E |
that were made for MPI/RT-1.0 at the last | | |
meeting) | | |
Tony, Leonard, Manoj | | |
| | |
19 Default Constructors | E | E |
Cui, Nathan, Shane | | |
| | |
20 New Instrumentation metrics | E-L | E-L |
(Add as needed with experience) | | |
| | |
21 More error returns | E-L | E-L |
Arkady, Tony | | |
| | |
22 Remove restriction of homogeneity on | E-M | 2/4/99 |
bufiter parameters: | | |
dataspec and bufsize for channel endpoints | | |
Shane, Arkady | | |
| | |
23 Unreliable Channels | M-L | 2/4/99 |
Leonard, Tony | | |
| | |
24 Fault Tolerance, Fault Handling, Fault | L | M |
Recovery | | |
| | |
25 Light-Weight Handlers | E | E |
Tony | | |
| | |
36 Half Channels (I/O) | M | 2/3/99 |
Shane, Tony | | |
| | |
37 Default Handlers | E | E |
Tony | | |
| | |
38 Group parameter scope for the commit | M | E |
Tony, Arkady | | |
| | |
39 Mechanisms for passing in externally build | M-L | M |
schedules to MPI/RT | | |
Arkady, Tony | | |
| | |
40 Logical grouping of event kinds | L | M |
(arithmetic on events) | | |
Arkady, Leonard | | |
| | |
41 PGCOs | M-L | M |
Arkady | | |
| | |
42 QoS for handlers (add deadlines) | M | E |
Arkady, Steve | | |
| | |
43 Mutable buffers (late binding of address) | E | 2/4/99 |
Steve, Robert Ginn | | |
| | |
44 Variable length buffers | E | 2/4/99 |
Steve, Robert Ginn | | |
| | |
45 Periodic triggers | M | E |
Steve, Leonard, Arkady | | |
| | |
46 Unique ID for running MPI/RT version | M-L | 2/4/99 |
Leonard, Arkady | | |
| | |
47 QoS for QoS (granularity for raising errors) | L | 2/5/99 |
Leonard | | |

MPI/RT Forum Minutes

Minutes for the Dec. 2-4 2022 MPI/RT Forum meeting held at MITRE in Bedford, MA.

Anthony Skjellum (chair) MSU
Arkady Kanevsky (chair) MITRE
Yogi Dandass MSU
Shane Hebert MSU
Steve Paavola SKY
Manoj Apte MSU
Zhenquian Cui MSU
Dennis Cottell SPAWAR
James Lebak MIT/LL
Leonard Monk CSPI
Nathan Doss LM-GES
Anna Rounbehler Raytheon
Sharon O’Neal Raytheon
Richard Games MITRE

Summary of official votes:

  1. Objects chapter: third and final formal vote - 9/0/2

  2. Types and Datatypes Chapter: third and final formal vote - 9/0/2

  3. Handler chapter: third and final formal vote - 8/0/3

  4. Event chapter: third and final formal vote - 7/0/5

  5. Buffer Management chapter: third and final formal vote - 10/0/2

  6. Channel overview chapter: third and final formal vote - 8/0/3

  7. Point to point channel: third and final formal vote - 9/0/2

  8. Collective Channel chapter: third and final formal vote - 9/0/2

  9. Channel Operations: third and final formal vote - 7/1/3

  10. Quality of service specification chapter: third and final
    formal vote - 7/0/3

  11. Committing Objects chapter: third and final official vote - 8/0/2

  12. QoS Overview Chapter: third and final formal vote - 6/0/3

  13. Instrumentation Chapter: third and final formal vote - 6/1/3

  14. The meeting came to order at 8:30 AM on December 2, 2022

  15. An etiquette of email discussion was briefly discussed.
    We decided to include “MPI/RT” on the subject line.
    Also we decided to reply only to the entire mailing list to
    reduce message duplication and to ensure that the whole discussion
    appears on the reflector.

  16. Institutions represented at the meeting: LMCO/GES,
    MSU, MSTI, Sky Computers, SPAWAR, MIT/LL, MITRE, CSPI, and Raytheon.

  17. A brief walk through of the small changes to
    the document posted for public comments was performed.

  18. The proposals that were discussed on the MPI/RT exploder were officially
    presented. Since less than half of the eligible institutions voted
    electronically on some of proposals and no official votes were taken
    electronically on others, the official votes were taken again on all
    the proposals.

A. Proposal 1: Container Iterators. There was a general agreement that
the proposal is good but it needed more details, especially for
“container_iterator”. It was unanimously decided to postpone inclusion
of this functionality until MPI/RT-1.1. Nathan Doss will provided an
updated proposal on the exploder (see MPI/RT-1.1 below).

B. Proposal 2: Buffer Iterators. This is the final resolution of the two
initial proposals that deal with performance issues for message
transfers and error checking. The following amendments were added:

  a. High performance implementations may choose to skip error
         detection code.  Implementations can offer a variety of
         optimization options to assist users in the debugging process.
      b. Added advice to users and implementers to reflect the first
         amendment.  An analogous statement should also go into the
         introduction chapter (chapter 1).
      The official vote for inclusion of this proposal as part of MPI/RT-1.0
      document passed 8/0/0. Chapters 5, 6, and 9 were updated with the
      proposal and presented on the second day of the Forum (Dec. 3) for the
      official (final) votes on the chapters.

C. Proposal 3: MPIRT_SLEEP. It was unanimously decided that this needed
further discussion and should be postponed until MPI/RT-1.1. For
it was decided to add advice to users to use system dependent calls
this functionality.

D. Proposal 4: 2-sided message transfers. There was a long discussion
centered around the states of the channel objects and the effect of
operations on the channel (start, activate, deactivate) and the state
the channel when a message transfer completes. As a result, the text
the proposal, and the state transition diagram for channels was
“Deactivate” on a the started channel is now disallowed. “Global
completion” and “global start” are no longer part of MPI/RT-1.0. Some
terminology related to the channel states was changed.

  The official vote on the modified proposal to be part of MPI/RT-1.0
      passed 7/0/0.
      Chapters 6 and 9 were updated with the approved proposal and presented
      on the second day of the Forum (Dec. 3) for official (final) votes.

E. The latex error in chapter 8 that forced the rest of the document to
be in italic was noted and fixed for the final official votes taken on
the second day of the Forum.

  1. Remove “Draft” from the title page.

  2. Preface: No formal vote required. The following changes were agreed
    a. Remove footnote.
    b. Add paragraph for thanks to institutions that hosted the Meetings:

  3. Minor changes were made to the Glossary.

  4. Introduction (chapter 1): final formal vote - 7/0/0. Amendments:
    A. Add a section 1.4 “What to expect in implementations”
    a. Two implementation libraries: high-performance and “debug” mode.
    b. Errors may not be detected in high-performance implementation.
    c. Erroneous programs produce undefined results.
    B. Move all the references to MPI into footnotes so the reader does
    not have to know MPI to read this document.
    C. Add section on the sidedness of message transfers.
    D. Add other advices.

  5. MPI/RT Objects (Chapter 2): final formal vote - 7/0/0. Amendments:
    A. Do not add default constructors in MPI/RT-1.0. Write a proposal
    for MPI/RT-1.1.
    only has only two returns MPIRT_EQUAL and MPIRT_NOT_EQUAL.
    [This change effects every object in the document and will be done
    globally for all chapters.]
    Write a detailed proposal for an expanded compare function for
    MPI/RT-1.1. Consider a special compare function for containers.
    C. Fix section headings for generic container and vector accessor
    in section 2.4.1.
    D. Add MPIRT_GROUP_SELF in section 2.5.

  6. Types and Dataspecs (Chapter 3): final formal vote - 7/0/0.
    Only minor editorial comments, a clearer definition of transmittable
    MPI/RT types, and the integer values for MPIRT_TRUE and MPIRT_FALSE.

  7. Event Delivery Abstraction and Handlers (Chapter 4): final formal
    vote - 5/0/1. Amendment:
    A. Synchronize the use of “kind of” event with “class of” events
    the document. Add the chosen term to the glossary.

  8. Buffer Management (Chapter 5 with the proposal 2 - see above):
    final formal vote - 9/0/0. Amendments:
    A. Remove “semi-opaque” from the figures.
    B. Add missing get/set operation for the new bufiter parameters.
    C. Add the following rules for a buffer (resolution of prolong discussion
    on when it is safe to read/write into memory of a buffer):
    a. A buffer object is available if it is committed, until it is
    (into a bufiter) and after it is removed from a bufiter, until it is
    inserted again.
    b. It is illegal to read/write to a location after commit,
    unless every committed buffer object whose memory region contains
    that location is available.
    c. It is illegal to insert a buffer unless it is available.
    d. It is illegal for a buffer object to be in the initial buffer list
    of more than one bufiter being committed.
    e. The implementation will not read/write a location unless
    at least one user-created buffer containing that location is
    into a bufiter.
    D. Add a buffer state transition diagram (which was agreed upon at the
    E. Come up with the better name for operations in section 5.3.
    F. Fix C++ binding for MPIRT_CVECTOR_CREATE_BUFFERS.
    G. Fix C++ binding for MPIRT_BUFITER_CREATE.
    H. Synchronize the consistent use of formats for S- and V-containers.
    I. Fix Bufiter state transition diagram.
    J. Fix old get/set operations for bufiter parameters to use new terms.

  9. Channel overview (Chapter 6 with the proposals 2 and 4 - see above):
    final formal vote - 9/0/0. Amendments:
    A. If MPIRT_ACCEPT_ANY is used for all channel endpoints then it is
    equivalent to MPIRT_BEST_EFFORT for all endpoints.
    B. Many editorial changes.

  10. Point-To-Point Channels (Chapter 7): final formal vote - 8/0/0.
    No amendments.

  11. Collective Channels (Chapter 8): final formal vote - 8/0/0.
    A. Remove “Shadow” from all advices to implementers.
    B. Change “implementation” to “example” in all advice to implementers.
    C. Fix latex italic problem.
    D. Editorial changes.

  12. Channel operations (Chapter 9 with the proposals 2 and 4 - see above):
    final formal vote - 8/0/0. Amendments:
    A. Change “Deactivated” state to “IDLE” state.
    B. It is illegal to “wait” on activated channel (for MPI/RT-1.0).
    (We will study this issue for MPI/RT-1.1).
    C. “Wait” on an idle channel returns immediately, without error.
    D. Channel should be an IN parameter for MPIRT_CHANNEL_WAIT and TEST
    E. Editorial changes.

  13. QoS Overview (Chapter 10): final formal vote - 8/0/0.
    A lot of editorial changes.

  14. QoS Specification (Chapter 11): final formal vote - 6/0/3.
    A. For Time driven channels:
    a. MPIRT_TIME_SPEC_IGNORE is not allowed as a start parameter.
    b. If start and deadline are specified as absolute, the time period
    In this case only one message transfer will take place.
    c. If period is specified (not IGNORE nor NULL) then start and deadline
    must be relative.
    d. Rename “start” to “phase.”
    e. Phase is less than or equal to the period.
    B. For event-driven channels:
    a. If the QoS spec is the same for all channel endpoints then the
    occurrence of the event transfers only one message over the channel.
    D. More detailed, and less restrictive specifications will be considered
    for MPI/RT-1.1.
    E. Editorial changes.

  15. Committing Objects and Resource Allocation (Chapter 12) - final formal
    vote 9/0/0. Amendments:
    A. It is the user’s responsibility to ensure that there are
    no operations pending when the commit function is called.
    B. The objects that were committed before the commit() call
    retain their states across the commit() call.
    C. Copy Figure 13.1 into this chapter.

  16. Initialize and Termination (Chapter 13) - final formal vote 9/0/0.
    No amendments.

  17. Clocks (Chapter 14) - final formal vote 9/0/0.
    No amendments.

  18. Instrumentation (Chapter 15) - - final formal vote 8/0/1.

  19. The final vote on the entire document as amended - 8/0/1.

  20. Reinsert the Function Names index.

  21. Add a section on “How to read the document” describing typographical

  22. Fix all C++ bindings.

  23. Resize figures to fit.

  24. Remove the references to MPI/RT-1.0 and MPI/RT-1.1 from the document.

  25. Change all the derivatives of the MPIRT_OBJECT_COMPARE function to
    the change to MPIRT_OBJECT_IS_EQUAL.

  26. The last day of the Forum was spent on discussion of MPI/RT-1.1 and
    beyond. It was agreed that for MPI/RT-1.1 we should just make small
    incremental additions to MPI/RT-1.0. As a topic is resolved (similar
    to the two proposals that where voted in during this meeting) and passes
    three formal votes, it will become a separate chapter of MPI/RT-1.1 and
    will be merged with MPI/RT-1.0.

    Below is the list of topics we are considering. For each topic we asked
    A. What is the expected time for completion?
    B. How quickly should we start discussing it?
    The possible answers are: E - early, L - later, and M - middle.
    E - answer for the first question means that it is part of MPI/RT-1.1.
    L - answer for the first question means that it is not part of
    M - answer for the first question means that we may have at least a
    of the topic resolved for MPI/RT-1.1.

    Here is the list of the topics discussed, the answers to the two
    and the person(s) in charge of the topic to ensure its progress.

TOPIC | Expected | Expected |
and person(s) in charge | Discussion | Completion|

1 Data-reorganization support | M | E |
Tony, James (from DATA REORG EFFORT) | | |
| | |
2 Collective operations (ALL_TO_ALL) | E | E |
Games, Arkady | | |
| | |
3 Dynamic Process Management, | E-M | E |
Parallel Client Server, etc. | | |
Tony | | |
| | |
4 Interoperation of MPI/RT Implementations | L | E |
Tony | | |
| | |
5 Support for Java binding of MPI/RT | L | M |
(other languages too?) | | |
Tony | | |
| | |
6 Full-QOS-based MPI/RT with mode changes | M | E |
(such as QOS capability for mode changes) | | |
Arkady, Steve | | |
| | |
7 Explicit ability to work with real-time | L | M |
schedulers for process scheduling | | |
Manoj | | |
| | |
8 Work to support k-slack channel mode | M | E |
(whereas MPI/RT 1.0 is 1-slack) | | |
Tony | | |
| | |
9 Support for IDL for MPI/RT | L | L |
Tony | | |
| | |
10 Fine grain, Coarse grain QOS specifications | M | M |
Arkady, Leonard | | |
| | |
11 More powerful data descriptor to allow | M | E |
strided and non contiguous buffers | | |
James, Tony | | |
| | |
12 how to handle other resources’ schedulers. | L | M |
Currently we have dealt with memory and | | |
network in two different ways and via | | |
completely different specs… | | |
(related to 7 and 10) | | |
Arkady, Leonard, Manoj | | |
| | |
13 Operations on containers. The old issues | E | E |
of propagating the operations to every object | | |
in a container (probably for the containers | | |
that contain homogeneous objects only). | | |
Nathan, Dennis | | |
| | |
14 Container Iterators | E | E |
Nathan, Dennis | | |
| | |
Arkady, Tony | | |
| | |
16 Channel state transition (old busy state) | E-M | E |
Cui, Arkady | | |
| | |
(related to 16) | | |
Arkady, Cui | | |
| | |
18 QoS Spec for channel (remove restrictions | E-M | E |
that were made for MPI/RT-1.0 at the last | | |
meeting) | | |
Tony, Leonard, Manoj | | |
| | |
19 Default Constructors | E | E |
Cui, Nathan, Shane | | |
| | |
20 New Instrumentation metrics | E-L | E-L |
(Add as needed with experience) | | |
| | |
21 More error returns | E-L | E-L |
Arkady, Tony | | |
| | |
22 Remove restriction of homogeneity on | E-M | E-M |
bufiter parameters: | | |
dataspec and bufsize for channel endpoints | | |
Shane, Arkady | | |
| | |
23 Unreliable Channels | M-L | E |
Leonard, Tony | | |
| | |
24 Fault Tolerance, Fault Handling, Fault | L | M |
Recovery | | |

Minutes for the 22 July 2022 MPI/RT Forum meeting held in Washington DC.

Anthony Skjellum (chair) MSU
Arkady Kanevsky (chair) MITRE
Jon Hiller STA
Joe Fogler UNM/AHPCC
Yogi Dandass MSU
Shane Hebert MSU
Steve Paavola SKY
Manoj Apte MSU
Zhenquian Cui MSU
Dennis Cottell SPAWAR
Randall Judd SPAWAR/TASP
David Coker ISI
James Lebak MIT/LL
Michael Greico ASEC
Leonard Monk CSPI
Nathan Doss LM-GES
Anna Rounbehler Raytheon
Robert Babb DU
Chris Hayden Sanders

Formal voting institutions:
Raytheon, DU, CSPI, Sanders

Summary of official votes:

  1. Objects chapter: second formal vote - 9/0/2

  2. Types and Datatypes Chapter: second formal vote - 9/0/2

  3. Handler chapter: second formal vote - 8/0/3

  4. Event chapter: second formal vote - 7/0/5

  5. Buffer Management chapter: second formal vote - 10/0/2

  6. Channel overview chapter: second formal vote - 8/0/3

  7. Point to point channel: second formal vote - 9/0/2

  8. Collective Channel chapter: second formal vote - 9/0/2

  9. Channel Operations: second formal vote - 7/1/3

  10. Quality of service specification chapter: second formal vote - 7/0/3

  11. Committing Objects chapter: second official vote - 8/0/2

  12. QoS Overview Chapter: second formal vote - 6/0/3

  13. Instrumentation Chapter: second formal vote - 6/1/3

  14. The forum thanked Jon Hiller and STA for hosting the forum meeting in
    Washington DC.

  15. June minutes passed - 11/0/3

  16. Glossary:
    a. Identify those objects that are defined by MPI/RT.
    b. Look through document for other undefined terms.

  17. Introduction
    a. Add a high level description of how the user sets up an MPI/RT
    b. Add an introduction for implementers.

  18. Objects chapter: second official vote - 9/0/2
    a. Add advice to user that objects they create should be freed.
    b. Explain MPIRT_OBJECT_TYPE
    c. Explain that there are MPI/RT types that are directly accessible
    by the user, as opposed to opaque MPI/RT objects, and a subset
    of these are transmittable.
    d. Rewrite overview section to be clearer.
    e. Several fixes to object figure - Add individual collective
    channels. Make figure landscape and larger.
    f. Object body description - add diagram to make clearer.
    g. Remove - section 2.1.1 header and possibly move to chapter 7
    just before QoS issues. Consider precis for chapter 1.
    h. All objects have names. However, certain objects do not need
    unique names, and can use NULL for the object names. Write
    description that commit can fail when the objects requiring
    unique names do not have unique names.
    i. The Object name does not have an impact on the compare function.
    j. Remove references to “caching.”
    k. Make “Attributes are scalar…” advice to users be part of
    regular text.
    l. Vector containers - remove Cvector_int. We now have a
    CVector_base object with descendants: CVector and Groups.

  19. Types and Datatypes Chapter: second official vote - 9/0/2
    a. Basic Datatypes are descriptors, not declarators.
    b. MPIRT_datatype are now objects. Added to hierarchy diagram.
    c. DATATYPE is now called DATASPEC.
    e. Type: Need MPIRT_INT64, with associated arithmetic operators.
    f. Mod for Int64 follows C/C++ rules.
    g. Move section 3.3 to section 3.1
    i. Remove DATASPEC_LONG_DOUBLE unless it is in the ANSI C standard.
    j. Time spec types are MPIRT_TIME_ABSOLUTE, MPIRT_TIME_IGNORE, and

  20. Handler chapter: Second Formal vote - 8/0/3
    a. Merge events and Handler chapters. Event chapter first.
    b. Make all tables have smaller fonts so they fit on the page,
    perhaps using vertical struts.
    c. Event Objects should now be called Control Channel/Control

  21. Event chapter: Second formal vote - 7/0/5
    a. Event class renamed to EvDel (Event Delivery System).
    b. Need a figure for event delivery abstraction
    c. Add exceptions for error codes in C++ bindings.
    d. Three proposals were considered.
    i. - 1: There is only 1 explicit receptor for event per
    2: The retrieve receptor retrieves the explicit
    3: QoS Chapter - Add group parameter to PrTiEv, PrEv, and
    Constrain object.
    ii. - 1: Receptors have only async handlers
    2: Multiple receptors per event name per process allowed.
    3: Objects can have async handlers (callbacks).
    4: Silent on implementation of “implicit receptors.”
    5: Extend Qos parameter for “implicit receptor.”
    iii. - 1: Multiple receptors
    2: Silent on order for async handlers.
    3: No retrieve function
    The following were voted on:
    A. proposal iii - 4 votes.
    B. proposal i with points 1 and 2 only - 4 votes.
    C. proposal i with point 1 and proposal iii with
    point 3 - 4 votes.
    e. The previous proposals were clarified by the following votes:
    i. Single explicit receptor per event name per process - 10/6/0.
    ii. Retrieve function only returns explicit receptors - 13/1/2.

  22. Buffer Management chapter: second formal vote - 10/0/2
    a. Create picture for MPIRT_CVECTOR_CREATE_BUFFERS to clarify.
    b. Figure: need to add transitions to/from uncommitted bufiter to
    partially full and full states.

  23. Channel overview chapter: second vote - 8/0/3
    a. Add “Introduction” section heading where missing (all
    b. Add QoS failure situations to state transition diagram.
    c. When the failure happens during a buffer transition, the
    implementation retries as long as QoS permits. When QoS failure
    happens, the buffer is moved to out-bufiter and the event is
    raised. Write an example demonstrating this situation.
    d. Add QoS BestEffort when channel doesn’t need QoS. Add text in
    the QoS Chapter.
    e. Make sure all channel conditions appear in this chapter.

  24. Point to point channel: second formal vote 9/0/2

  25. Collective Channel chapter: second formal vote: 9/0/2
    a. Fix figure 9.1
    b. Write short descriptions of what each operation means.

  26. Channel Operations: Formal vote - 7/1/3
    a. Fix figure 10.1
    b. Channel_Test() returns MPIRT_CHANNEL_BUSY and MPIRT_CHANNEL_IDLE.
    c. Proposal: Channel state transition has 2 states idle and running.
    Two conditions channel_started and channel_completed.
    Passed - 9/3/3
    d. Proposal to delay formal vote until text is modified to reflect
    the changes in the chapter was defeated - 3/9/0.

  27. Examples moved to appendix.

  28. Quality of service specification chapter: Second official vote - 7/0/3
    a. The channel QoS specification for QOS_CHANNEL_PREV and
    QOS_CHANNEL_PRTIEV lacks the QoS sepcification for the implicit
    receptors. In order to correct this, the following functions will be
    added to start_event for QOS_CHANNEL_PREV and QOS_CHANNEL_PRTIEV:
    start_scope(group), and start_qos(qos_receptor). For stop_event
    for QOS_CHANNEL_PREV the stop_scope(group) and stop_qos(qos_receptor)
    will be added. The corresponding accessor operations will also be

  29. Committing Objects chapter: second official vote - 8/0/2
    a. Commit - Each object is committed once.
    b. Explicitly state that all objects do not need to be explicitly listed
    in commit container. Implicit relationships, for example bufiter for
    channels, will be committed automagically.

  30. QoS Overview Chapter: second formal - 6/0/3
    a. EvDel needs to be reflected in this chapter.
    b. Conditions for channels need to be added to this chapter.

  31. Instrumentation Chapter: second Formal vote - 6/1/3
    a. Replace MPIRT_String with String_name.
    b. Write explanation for minimum set of metrics table.

  32. Proposal to keep container operations in version 1.0 defeated - 5/6/2.

  33. Init-term chapter:
    a. Friendly amendment - add function and constant to return version number
    of MPI/RT.

Draft Minutes for the June 23-25 2022 MPI/RT forum meeting held at Mississippi State University


Anthony Skjellum (chair) MSU
Arkady Kanevsky (chair) MITRE
Darwin Ammala MPI Software Tech
Steve Paavola Sky
Dennis Cotttel SPAWAR
James Lebak MIT/LL
Zhenquian Cui MSU
Greg Henley MSU
Nathan Doss Lockheed Martin-GES
Marc Fidler Sanders
Robert George MSU
George Crawford III MPI Software Tech
Yogi Dandass MSU
Michael Grieco ASEC/NSA
Manoj Apte MSU
Shane Hebert MSU
Eric Holbrook MPI Software Tech
Clay Taylor, Jr. MPI Software Tech
Thomas McMahon MPI Software Tech
Mark Lin MPI Software Tech
Robert Babb DU
Gabriel Jones MSU

Summary of official votes:
Handlers (chapter 4): first official vote - 9/0/1
Event objects (chapter 5): first official vote - 11/0/0
Object Decoration (Section 2.3): first official vote 11/0/0
Containers (section 2.4): first official vote 11/0/0
Groups (section 2.5): official vote 11/0/0
Datatypes (Chapter 3): first official vote 11/0/0
Handler QoS (Section 14.2): first official vote 11/0/0
Event QoS (Section 14.3): first official vote 9/0/0
Initialization and Termination (Chapter 15): second official vote 9/0/0.
Instrumentation (Chapter 17): first official vote 9/0/2

  1. The meeting came to order at 8:30 AM on June 23, 2022

  2. The minutes from the previous meeting was discussed and approved
    after agreeing that the vote recorded in the minutes for the QoS
    chapter was limited to the QoS specification for channels.
    Approved: 16/0/1

  3. Institutions represented at the meeting for official votes:
    Sanders, Lockheed Martin/GES, MSU, MPI Software Tech, Sky, SPAWAR,

  4. A brief walkthrough of the current document was performed.

  5. Handlers (chapter 4): official vote - 9/0/1
    a. If the object that triggered an event is not local to the
    handler function (in the same process), the handler function
    object and condition parameters respectively.
    b. Handlers may be synchronous or Asynchronous. Added a flag in
    the constructor to reflect this. Also add an accessor function
    to retrieve/set the flag.
    c. First the asynchronous handlers are present to the scheduler,
    then the synchronous handlers are started.
    d. Move light weight operations that generate upcall handler calls
    (section 4.2.2) to MPI/RT 1.1.
    e. A recoverable event delivers the results of the synchronous
    handlers for the local receptors back to the MPI/RT object that
    raised the event trigger indirectly. The implementation uses the
    return value from the handler function to determine if a retry of
    the operation which caused the recoverable event can be performed.
    f. Lightweight handlers are implemented as functions.

  6. Event objects (chapter 5): official vote - 11/0/0
    a. The “event” prefix in the LIS specification parameters for
    certain functions was removed. For example, event_qos and
    event_scope in MPIRT_TRIGGER_CREATE are now qos and scope,
    b. Users can create triggers with the same names as are used for
    the system generated implicit triggers.
    c. The proposal to make the MPIRT_TRIGGER_RAISE function
    asynchronous approved by straw vote - 20/0/2.
    d. The proposal to allow at most one receptor for a given event
    name per process (regardless of handler type: synchronous or
    asynchronous) was approved by straw vote - 17/0/2.
    e. The proposal to add a function to retrieve the receptor
    associated with the event name in the local process
    was approved by straw vote - 9/4/6.
    f. Raising an event is a non-blocking operation.

  7. MPI/RT Objects (Chapter 2)
    a. Attribute Caching changed to Object Decoration (Section 2.3) -
    official vote 11/0/0
    i. Change “copy” in function names and bindings to “dup”.
    ii. Remove Callback from the object hierarchy diagram since it
    is only a function, not an object.
    iii. change DECOR_KEYVAL in LIS to KEYVAL.
    iv. Add LIS for functions in the section.
    v. Add datatype of “choice” for void * parameters in LSI
    vi. Clarification was made that duplicating a KEYVAL creates a
    new object with same callbacks, but a unique ID.
    vii. Added accessor (get/set) functions for KEYVAL callbacks.
    viii. Removed the MPIRT_KEYVAL_FREE function because it is
    available at the MPIRT_OBJECT level.
    b. Container (section 2.4) - official vote 11/0/0
    i. Shorten object names to MPIRT_CSET, MPIRT_CVECTOR,
    ii. Proposal to keep “syntactic sugar” (functions of
    convenience, but not essential to the standard) such as
    union, concatenate, difference etc was defeated by straw
    vote - 0/13/5.
    iii. Proposal to keep vector functions incl, excl was rejected
    by straw vote - 1/13/4.
    iv. Function to compare containers was removed because it
    is common to all objects. The comparison function should
    be at the MPIRT_OBJECT level.
    v. The proposal to keep function TRANSLATE_INDICES was rejected
    by straw vote - 0/16/2.
    vi. Removed _ELEMENT from LSI bindings for all containers. For
    vii. Move common functions to the top of the chapter (just as in
    other chapters/sections).
    c. Groups (section 2.5) - official vote 11/0/0
    i. Group containers must perform automatic compaction when
    elements (process addresses) are removed, ensuring there
    are no “holes” in a group container.
    ii. GROUP_WORLD is read-only.
    iii. Containers for group world are identical on all processes
    (that is, the order of task_addresses is the same in all of them).
    d. Miscellaneous Types (section 2.6) official vote: 11/0/0

  8. Datatypes (Chapter 3) - official vote 11/0/0
    a. Move section 3.2 to chapter 2 and rename as “Miscellaneous Objects”
    Miscellaneous objects are: MPIRT_STRING_NAME, MPIRT_TASK_ADDRESS.
    b. MPIRT_STRING object was removed. It is now a typedef to char*.
    c. Need to create a table of serializable MPI/RT objects that can
    be transmitted.

  9. Quality of Service (Chapter 14)
    a. Handler QoS (Section 14.2) - official vote 11/0/0
    i. QoS for handlers can only be priority and timeout.
    It was decided that QoS should only support objects that are
    created/maintained by the implementation. Since handlers are
    just user written code, the implementation does not know how
    long it takes to execute, nor any hidden dependencies (data
    dependencies etc.).
    b. Event QoS (Section 14.3) - official vote 9/0/0

  10. Instrumentation (Chapter 17) - official vote 9/0/2
    a. Elapsed time in MPI/RT Metrics (Section 17.2) was changed to timer.
    b. The concept of attaching a probe to a user object was removed.
    c. In MPI/RT User Metrics (Section 17.4) type_size was changed to
    data_size, where data_size is the number of data_types in the
    d. MPIRT_PROBE_RETRIEVE_DATASIZE function was added to section 17.3.
    e. Probes work only on local MPI/RT objects.

  11. Initialization and Termination (Chapter 15) - official vote 9/0/0.
    a. Added advice to users that implementations can modify argc and
    argv parameters.
    b. Minor changes to figure to show that the commit operation, while
    the application is in a real-time mode, causes the application to
    go into a non-real-time mode until the commit operation succeeds.

DRAFT Minutes of the MPI/RT Forum Meeting of May 19-21, 2022

Name/Organization e-mail
Leonard Monk, CSPI
James Lebak, MIT/LL
Manoj Apte, MSU
Dennis Cottel, SPAWAR
Zhenquin Cui, MSU
Yogi Dandass, MSU
Nathan Doss, Lockheed-Martin/GES
Marc Fidler, Sanders
Robert Ginn, NAWC, AD
Chuck Browne, NAWCAD
Shane Hebert, MSU
Randy Judd, SSC-SD, TASP
Arkady Kanevsky, MITRE [co-chair, host]
Anthony Skjellum, MSU [co-chair]
Clay Taylor, MPI Soft. Tech.
Steve Paavola, SKY
Anna Rounbehler, Raytheon
Chris Hayden, Sanders

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, SKY, MITRE,
LM GES, Raytheon, Sanders, NAWC, MPI Software Technologies

Official vote summary for meeting (chapter numbers from May 18 version of the document:

Straw votes [selected]

The meeting came to order at 8:30am on May 19.

  1. The minutes of the previous meeting were approved. Official vote 14/0/2.

  2. The meeting opened with a discussion of the newest draft, which was distributed
    at the meeting, and put on the reflector as of May 18. A walk through
    of the document was offered, with highlights and lowlights offered.
    It was noted that several new chapters have appeared, several
    have been moved, while others rewritten and “cleaned up” for reading.
    The biggest changes are:
    document is split into two books: Standard and Best Engineering Practices,
    introduction for users,
    container section of MPI/RT objects,
    parallel Programming Models replaced group, communicators and context chapter,
    datatypes, light-weight handlers and upcall handlers,
    event objects, commitable objects, completable objects and QoS specification.

  3. Reading of “Parallel Programming Models and Application Formulation”
    chapter #5 (old groups, communicators and context chapter) was undertaken.
    – The groups replaces communicators as the scope defining entity.
    The primitive communicators were discussed that allows to have
    a very restricted MPI-1 type communication for setting up real-time
    a. The PCOMM (primitive communicator) proposal was voted down;
    straw vote 17/0/0
    The main reason for that is the that allows two different ways
    to do message transfers: channel way and send/receive way.
    b. The group operations were discussed. Since groups are containers
    it was decided that the group operation of this chapter should
    be moved into container section of the MPI/RT objects chapter
    and generalized for vector containers.
    c. There are no non-local operations on the groups.
    d. There is only one group specific operation (not a vector container
    operation) - GET_RANK.
    e. The remainder of this chapter (about one page) became
    a section of the MPI/RT object chapter.

  4. Reading of the Event Object chapter was conducted.
    The chapter was accepted with official vote of 7/1/0 with the following
    a. There is no need for the handlers for individual objects.
    The only one needed is for event receptors. That means that
    if users (application or implementation) would like to invoke
    a handler on some object condition then can create an event
    receptor for that object condition name. (The handler chapter
    was not read during this meeting so the effect on up-call handlers
    is not clear!)
    This change does not effect this chapter in itself but it means
    that handlers are removed for all objects except event receptors.
    b. Move a paragraph on MPIRT_NAME_NULL into object chapter.
    c. Create a separate section in the chapter for common event functionality
    (get, set) away from triggers and receptors sections.
    d. Put back the disappeared TRIGGER_RAISE operation.
    e. Change the names of Receptor operations from GET/SET for FLAG to
    IS_ENABLE and IS_DISABLE respectively.

        The updated chapter was distributed on May 21.
  5. Reading of the Buffer Management chapter was conducted.
    The chapter was accepted with official vote of 8/0/1 with the following
    a. Remove most of the advice to implementors in the intro section
    and move the remainder of it after buffer_create operation.
    Delete text prior to buffer_create in buffer object section.
    b. Change the bufset section completely. The new functionality
    should support buffer partitioning:
    Create container of buffers by chopping a “big” buffer
    into number of “small” buffers preserving the datatype and by creating
    a buffer for the reminder of the “big” buffer.
    Create a buffer that starts at specified displacement and of
    specified length in datatype items.
    All containers have names.
    The containers for buffers should be container vectors.
    Add advice to implementors on special memory and continuous memory.
    c. The buffer iterator section should be modified. Get rid of buflabels
    as a separate parameter for the iterator. Add two operations to
    assign a label for a buffer and to retrieve a label of a buffer.
    All buffers have default labels on commit.
    Delete all other operations that deal with labels and remove
    labels from buffer iterator accessor operations.
    Add text that all buffers in the bufiter are of the same size and
    datatype and not necessarily from the same container.
    Buffers are not placed into bufiter until commit.
    Add advice to implementors and examples.
    Fix the figure.

  6. A short discussion of the datatype chapter took place. It was unanimously agreed
    to keep basic datatype section plus add C++ mapping and MPI/RT specific datatypes.
    Remove the rest of chapter. Split MPI/RT datatypes into ones that are transferred
    in a message and the ones that are not. Create a separate non-transmittable datatypes
    section for the later.

  7. The Channel overview chapter was read. It was voted in with official vote
    of 8/0/0 with the following changes:
    a. Move rationale to point-to-point channel chapter.
    b Beef up descriptions of channel attributes
    c. Add “attributes” to the name of the section for common operations.
    d. Add forward reference to point-to-point and collective channels
    e. Synchronize the use of the names with point-to-point and collective
    channels chapters.
    f. Percolate the changes for event object chapter (dropping handlers)
    to this chapter.

  8. The point-to-point channel chapter was read and voted in with official vote of 7/0/0.
    a. The main changes deal with a name that is used as a generic
    object names, for the matching of channel endpoints and for channel
    event names. After long discussion and a lot of work we agreed that
    there is only one name used.
    For events the name implicitly generated using the following rule
    by concatenating parts separated by “:”:
    group name (not explicitly specified since group is an input parameter),
    local rank, remote rank, channel name, and condition name.
    The user specified name is the same for both ends of the channel.
    b. Move all text after the create operation into channel overview chapter.
    c. Add examples for the use of event names in the channel overview chapter.
    d. Drop “MPIRT_CHANNEL_TRANSFER” for the name of a channel condition.
    e. Percolate the changes for event object chapter (dropping handlers)
    to this chapter.

  9. The Collective channel chapter was read and voted in with official vote of 7/0/0 with
    the following amendments:
    a. Percolate all changes for Point-to-Point channels and channel overview.
    b. Combined three tables of reduce operation into one and add C++ datatypes.
    c. Synchronize with Commitable chapter.
    d. Add bufiter with zero length buffers for barrier operation.
    e. Update figures and their descriptions.

  10. The commitable object chapter was read. While the chapter was not officially
    voted on the following vote was taken after long discussion
    and debate. There is a single commit/uncommit operation instead of
    a three phase current proposal. Straw vote 8/7/0.
    The old admit/transit proposal with discussed variation of separate
    or combined Transit and Resource objects will be reconsidered for MPIRT-1.1.
    Update figure for Init/Term chapter to reflect this decision.

  11. The completable chapter was read and voted on. The official vote is 7/0/0
    with the following amendments to the chapter:
    a. Change Completable to Channel.
    b. Merge this chapter with Data Transfer chapter and rename it
    operations on the channels.
    c. Change input parameter to allow operations on a channel and on a
    container of channels.
    d. Change discussion to a regular text.
    e. Merge discussion after test operations with the text for the figure
    for state transition diagram for a channel.
    f. Synchronize the use of the status for the test with the state in
    state transition diagram. Update channel state transition diagram
    and define events as transitions between the states. Status represents
    a state.

  12. The data transfer chapter was read and officially voted in 7/0/0 with the following
    a. Add QoS error to the condition list, update the State transition diagram
    b. Synchronize with channel chapters and event chapter (no more handlers)
    c. Merge with completable chapter (before examples).
    d. Convert discussion into advice to users.
    e. Add C++ bindings.
    f. Add pointers to QoS chapters.
    g. Split examples into two subsection on for time-driven model and
    one for event-priority driven one.
    h. Update all examples and all example of the next chapter.

  13. Several chapter and sections of the Real-Time Programming models and QoS.
    – The names of all the communication paradigms chapter should be uniform.
    – The section on Time Specification of the time-driven paradigm was
    officially votes in with the vote of 9/0/0. The only amendment was
    reflect that it is not a datatype to be transferred in a message.
    – The Event-Driven Chapter was voted in by official vote of 7/0/2 with the
    following amendments:
    a. Move MPI references to footnote,
    b. Change Comm to Group,
    c. Change the name of the event as defined for the channels,
    d. Add the example event names
    – The Priority-Driven chapter was officially voted in with the vote of 9/0/0.
    – Part of the previous vote was also to merge chapter 17 through 20 into
    one chapter. This will leave this part with two chapters: one describing
    paradigms and one for specification.
    – The QoS Specification chapter was officially voted in with the vote of
    7/1/1 and with the following amendments:
    b. Change Receptors to Event name,
    c. Work out a proposal on adding time-spec for events (allow deadline
    instead of timeout).

The meeting concluded at 12:15pm on May 21.

DRAFT Minutes of the MPI/RT Forum Meeting of April 8-10, 2022

Name/Organization e-mail
Manoj Apte, MSU
Robert Babb, Denver U.
Dennis Cottel, SPAWAR
Zhenquin Cui, MSU
Yogi Dandass, MSU
Nathan Doss, Lockheed-Martin/GES
Marc Fidler, Sanders
Robert Ginn, NAWC, AD
Mike Grieco, ASEC/NSA
Shane Hebert, MSU
Randy Judd, SSC-SD, TASP
Arkady Kanevsky, MITRE [chair]
Tony Lee, NAWCAD
Steve Paavola, Sky
Anna Rounbehler, Raytheon
Alan Skillicorn, MITRE

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, Sky, MITRE,
LM GES, Raytheon, NSA, Sanders, NAWC

Official vote summary for meeting:

Straw [selected]:
MPIRT_OBJECT_TYPE - the name of the enumerator (replaces all
of it). passed 13/0/1

The meeting came to order at 8:30am on April 8.

  1. Tony is sick and not able to attend the meeting.

  2. The meeting opened with a discussion of the newest draft, which was distributed
    at the meeting, and put on the reflector as of April 7. A walk through
    of the document was offered, with highlights and lowlights offered.
    It was noted that several new chapters have appeared, several
    have been moved, while others rewritten and “cleaned up” for reading.
    The biggest changes are:
    glossary has most of the terms now,
    profiles and resource constrained chapters are moved from part I to
    part VI,
    MPI/RT object chapter is updated with new container section,
    attribute caching section moved from communicator chapter, and
    new UML object hierarchy diagram (appeared at the end of the document
    due to the problems with postscript),
    datatype chapter was copied from MPI-1,
    light-weight handler section (created during the meeting),
    committable objects (to be renamed Resources) outline created,
    completable objects chapter written,
    channel chapters updated,
    transfer examples updated,
    all chapters of QOS rewritten to reflect February meeting votes,
    initialization and termination chapter rewritten,
    instrumentation (was distributed during the meeting),
    resource constrained MPI-1.0.

  3. Shane presented Inertia. Inertia 0.5 is ready and available.
    MSU also supplied several MPIRT example that run on it.
    Yogi will create a link from MPIRT web site to it, and
    will add examples into MPI/RT document.

  4. Reading of the “Objects” chapter #4 (old chapter 5) was undertaken.
    It was read every day with an updated version provided by Arkady, Shane
    and Yogi dayly.
    – The new UML diagram is separately presented at the end of the
    After a long discussion on classes and names following resolutions
    were made on the first three sections of the chapter:
    a. Drop “object” from user_object and system_object on
    the diagram.
    b. Remove User and System objects ancestors from the diagram.
    (this was done on the second day which explains redundant
    previous resolution.) straw vote passed 7/2/3.
    b. Make the use of object class, object, object type and
    object instance consistent through out the document.
    c. Drop MPIRT prefix from diagram (it only appears at the root).
    d. Remove the clock as a singleton object from the diagram
    completely. This gets rid of the only singleton object and
    document no longer need to describe singleton objects.
    (it may come back if we decide to have SYSTEM object and
    update function names accordingly (like MPIRT_SYSTEM_INIT).
    It was discussed at the last day of meeting but no decision
    was made.)
    e. Define a virtual DUP at an ancestor level (old MPIRT_OBJECT).
    C does not need to have a separate dup on each object level,
    while C++ has virtual inherited dup.

– The generic operations on all objects were discussed in part
as a container discussion (section 4.4).
Following resolutions were made:
a. MPIRT_OBJECT_TYPE - the name of the enumerator (replaces all
and other variations of it). straw vote passed 13/0/1.
b. Add operation MPIRT_OBJECT_GET_OBJECT_TYPE that is enhereted
by all objects (done during the meeting. included in the
official vote).
c. Add definitions of IN and INOUT parameters to the glossary.
d. All objects that are used as parameters are passed and
stored as references. The corresponding get_parameter call
returns the stored reference. The underlying object identified
by the reference can be modified (which effects all objects
that reference it) unless it is committed. (itself or via
some committed object that references it). As soon as
an object is committed its parameter objects are committed.
Official vote passed 9/0/1.
e. The user does not need to “free” the reference he/she obtained
from the get operation (see previous item). However, it is
user responsibility to free the objects that are no longer
needed and are not in use. A single free destroys
uncommitted/noncommitted object. Free is shallow and does
not work on committed objects. Straw vote passed 9/1/5.

        Formal first vote on section 4.4 passed 7/0/0.

– The first reading of container section (4.6) took place (actually
there were three readings of it, one per day). The following
changes were agreed upon, made during the meeting and voted in:
a. Remove “size” from Container_set.
b. Add “name” parameter to container object (every MPIRT object
has a name (of type MPIRT_STRING_NAME), and a type
c. Add “count” attribute for a container (both set and vector
types). Straw vote passed 13/0/2.
d. Containers do not support built-in datatypes.
e. Define homogeneous. Remove wording that containers are
homogeneous. If a container contains objects of different
types one can walk up the object hierarchy to the topmost
level in the worst case to get every object “homogenized”
to MPIRT_OBJECT in the worst case.
f. Remove “generic” container object and split containers into
sets and vectors.
g. Return error if element to be removed is not in the set, or
currentelement for retrieve_next is not in the set.
h. Add function to query of a reference for an
element is in the set.
i. Return error if duplicate reference is trying to be
inserted into a set. Straw vote passed 10/1/1.

        Formal first vote for the section 4.6 passed 8/0/1.
  1. The event chapter #8 was read for the first time. This is a new chapter
    that presents events: triggers and receptors as agreed at the last meeting.
    The discussions took a long time slipping into QOS topics and into
    communicators discussions. No formal vote was taken due to the flex of
    communicator/group chapter that is needed to define scope of events.
    Never the less, the following changes were agreed upon:

          a. Triggers and receptors need generic object names (as all

MPIRT objects (local), and they need a globally unique name
for matching two sides of an event (trigger/receptor).
The event name is used for the second one.
b. The scope of the event delivery need to be reconsidered.
Existing groups and communicators definitions are not sufficient
for it.
c. Two events with the same QOS should be delivered in
the pairwise order. (If they are generated in some
rank and are delivered to the receptors of identical rank.)
This has no effect on this chapter but will be considered
at the next meeting when we will read QOS for events.
d. Add functionality to enable/disable receptors.
Disable receptors drop incoming corresponding events.
No queuing events on disable receptors.
Default value is enable.
This creates one more parameter for receptors and
one more function need to be introduced to get this
parameter (enable/disable status).
It is an error to enable abled receptor, and it is an error
to disable disabled receptor.
Straw vote passed 13/0/2.
e. Allow handlers for the receptor condition: receptor’s event
arrived. Straw vote passed 13/0/2.

  1. The handler chapter #7 was read for the first time. The major changes
    since last time is introduction of light-weight handlers and use
    containers in parameter definitions. No formal vote was taken due to
    the fact that people wanted more time to absorb changes and see all the
    places where handlers are used. The following changes were agreed upon:

         a. System handlers need object name.
             b. Standard needs to define default behavior for conditions

for which there are no handlers.
c. There was a long discussion on two types of handlers,
one which is recoverable (support callback) and which is not.
The call back version is needed to support bufiter overflow
and overflow condition where the same insert/remove buffer
operation need to be repeated after the successful return
of a handler. Shane and Steve will write a proposal for
the next time. Steve proposed at the meeting a skeleton
of the handler implementation; Yogi will add it to the
d. Details of the light-weight handlers are needed (current
text just provides APi). Example of its use for instrumentation
is needed.
e. Rename Object_type and Object_cond parameters
into anchor_type and anchor_cond in order to avoid function
name conflict with MPIRT_OBJECT_GET_OBJECT_TYPE that returns
the type of the calling object vs. the object that handler
is attached to.

  1. The initialization and termination chapter #23 was read and voted for
    the first time. Here are the list of changes that were made during the
    meeting and voted on at the last day:
    a. Add arguments definitions to function declarations
    (argc and argv). (done)
    b. Include advice to implementors about timeout. (done)
    c. Define the state after finalize and before init.
    Which processes are still alive. This takes care of an
    MPI problem when MPI_INIT create processes and
    MPIRT_FINALIZE destroys them. (done).
    d. Update the first figure to show process before init, after
    finalize, and add commit/transit operations to show
    synchronization points before any user data transfers and
    after it. Delete the second figure.
    e. Add definition of the application process.

    Chapter was formally votes in (first vote) 9/0/0.

  2. As part of the resource constrain chapter the following global decision
    was made:
    a. Split the document into two volumes: volume I -
    Standard - parts I-V,
    and volume II - parts VI - VII.
    No vote was taken but decision was unanimous.

  3. The discussion on Resource constraint MPI-1.0 took place.
    No formal votes were taken but many straw vote to guide the authors.

         a. There will be only one resource constrained profile for MPI-1.0

    b. Remove MPI_Errhandler functionality. Keep MPI_ERROR_CLASS.
    Remove now unnecessary error codes.
    c. Keep MPI_COMM_DUP and SPLIT. (29.2)
    d. Remove groups, COMM_CREATE, Inter_communicators, attribute
    caching, collective computations with extended datatypes,
    user defined collective computations, buffered send mode,
    sendrecvreplace, error strings, process topologies, derived
    datatypes (29.6, 29.7, 29.8, 29.10, 29.11, 29.12, 29.13,
    29.14 and 29.15).

  4. There was a discussion on the instrumentation chapter. No formal or straw
    votes took place. Some agreements were reached:
    a. Put the metric for 25.2 into a table.
    b. Remove operations on name. They are inherited from MPIRT Object.
    c. Rename operations. Get and Set are used only for object
    parameters. Use retrieve for other gets for this chapter.
    d. Consider rewriting 25.5 (generic probe) and moving it to
    e. Update API to reflect OO terminology used through out the

  5. Thanks were offered by the committee to Al Skillicorn for hosting this
    MPI/RT Forum meeting.

The meeting concluded at 12:45pm on April 10.

DRAFT Minutes of the MPI/RT Forum Meeting of February 24-26, 2022

Name/Organization e-mail
Darwin Ammala, MPI Softtech
Manoj Apte, MSU
Robert Babb, Denver U.
Dennis Cottel, SPAWAR [local host]
Yogi Dandass, MSU
Clair Guthrie, Navsea
Shane Hebert, MSU
Randy Janka, GTRI
Randy Judd, SPAWAR
Arkady Kanevsky, MITRE [co-chair]
Tony Lee, NAWCAD
Steve Paavola, Sky
Tom Robey, Khoral Research
Anna Rounbehler, Raytheon
Brian Schott, USC/ISI
Tim Singleton, Navy
Anthony Skjellum, MSU [co-chair]
Rick Steinberger, LM GES
Clay Taylor, Jr, MPI Softtech

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, GTRI, Sky, Denver U., MITRE,
MPI Softtech, LM GES, Khoral, Raytheon

Official vote summary for meeting:

The meeting came to order at 8:30am on February 24.

  1. The meeting opened with a discussion of the newest draft, which was distributed
    at the meeting, and put on the reflector as of February 24. A walk through
    of the document was offered, with highlights and lowlights offered by both
    co-chairs. It was noted that several new chapters have appeared, and several
    have been “cleaned up” for reading.

    The glossary was discussesd and augmented as part of this.

  2. The customary discussion of the TASP schedule, and MPI/RT’s progress was opened,
    with emphasis coming from Tim Singleton about the need for prompt conclusion of
    version 1.0. Skjellum and Kanevsky reiterated the positive progress. The TASP
    need for Inertia was also discussed informally, but the longer discussion was
    deferred to the Wednesday afternoon slot, in which Hebert would discuss Inertia.

  3. The 2nd reading of the “Clocks” chapter #21 was undertaken, with some minor issues
    raised, and then it was passed by formal vote of 7/0/0 (motion carried unanimously).

  4. 1st Reading of the “Objects” chapter #6 was undertaken.
    – Section 7.6 is to be put into 6.4
    – Various editorial fixes were offered; co-chairs suggested that they will
    do English editing later, and get help from an English-editorial person
    at the near conclusion of the effort.
    – Besides grammatical flaws, various paragraphs were promised to be rewritten
    for clarity.
    – It was agreed that Figure 6.2 would be redrawn in UML [one of Tony’s graduate
    students is working on UML for MPI/RT, so Diane Wooley will be on the
    hook for this. Randy Janka also agreed to try something.]
    – Despite modifications, Paavola moved and Kanevsky seconded (Skjellum was
    chairing at this point) the proposal to take 1st vote on Ch 6, minus section
    6.4, which will be reviewed and voted separately. The vote was 8/0/0 to
    accept the 1st reading modulo section 6.4 (unanimous).

  5. The committee in general asked that the editors depart from the standard headings of the
    report style, and introduce these readability features, to which the editors promised
    best effort:
    – Page numbers at center bottom,
    – headings left facing to improve finding out what is where

  6. The first review of the Communicators/Groups chapter was undertaken, which had already
    been modified to remove attribute caching to the Objects chapter.

    – It was noted in this context that the new 6.4 (not covered under topic 4 above),
    would apply to all children of MPIRT_Object
    – the callbacks would be made commensurable with MPI/RT handlers; it was noted
    that the original 1.0/1.1 version of certain MPI callbacks are flawed or
    problematic, and that this would be reviewed carefully.

  7. Some discussion of VSIP, streams, and p. 85 of document were offered. This led to
    no specific conclusion, other than that it would be revisited on March 3-4 at the
    Boston VSIP meeting, where the issues/modifications to VSIP would be covered.

  8. Points about content-addressibility of handler access was brought up by Paavola.
    Paavola wants to be able to “look up” handlers according to objects they attach to,
    and conditions. Paavola was asked to write this.

  9. It was agreed that all references to MPI-1 and MPI-2 would be put in descriptive
    footnotes, rather than in running text, so that it would not be necessary to read
    either or both of the MPI standards prior to learning MPI/RT. The rationale for
    the footnotes remaining would be to offer perspective to those who already know
    MPI, or who wish to make comparisons.

  10. Point-to-point and collective “Channels” chapter #12 was read.
    The following agreements (straw votes) where reached: the old note that qos spec for
    two (or more for collective case) endpoints of the channel must be the same. The only
    exception is a special constant MPIRT_QOS_TRANSFER_ACCEPT_ANY that is allowed to match
    any specification of the qos from another end(s) of the channel.

    The discussion on “strings” and names took place. We aggreed to have a datatype
    MPIRT_NAME to avoid space allocation for the strings and leave this issue to implementation.
    The name will still be INOUT parameter. These decision affected several operations
    in this and other chapters including earlier and later chapters: event-driven, and
    handlers. The part of the object hierarchy will apparently look like:


    Where the MPIRT_String datatype may be used more generically elsewhere.

  11. The straw vote was taken (passed by acclamation) to combine API to
    the highest object in the hierarchy to reduce the number of functions in the document
    as well as the size of the document. The point-to-point channel and collective channel
    will be combined into a higher level object (name “channel” will be used for that object
    and “ptchannel”) to accomodate that.

The meeting adjourned at 5:30pm on 2/24/98
The meeting reopened at 8:00am on 2/25/98

  1. Chapter 15, QoS, was read and voted on 9/0/1 (formal vote passed).
    Only minor rewording was suggested for this chapter.

  2. Chapter 16, Time Driven, was read and voted (with exception of section 16.3) in 10/0/0.
    Only minor rewording was suggested for 16.1 and 16.2. A portion of 16.3 that deal with
    definition of MPIRT_TIME_OBJECT datatype will be moved into datatype chapter. The
    remaining portion will be completely reworded. The new name will be chosen for the

  3. Chapter 17, Event-Driven, was read.
    After prolonged discussion the following decisions were voted on.

    There will be two committable objects: triggers and events (the names will be changed:
    one approach, Event will be general naming; what used to be events will be triggers;
    what used to be triggers will be ‘receptors’… this will be explored further and

    Each of the these have their own qos specification that is part of their specification.
    Events also assign names that are also used for trigger specification to match with
    the events. Formal Vote 10/0/3 (passed) on the functions/objects/behavior.

    The commit (for channel sets) will be “combined” with event registration.
    Also the same operation will “commit” other “commitable” objects including
    channels via channelset, events and triggers (combined object will be used for that),
    and “persistent generalized completable objects”. Vote 11/0/2

    The editors will present the new operations (in the new committable objects chapter)
    in the next version which is due at the beginning of the second week of March.

  4. During the event discussion, Chapter 19 was also discussed. The previously mentioned
    votes impact this.

  5. It was requested that the API not specify the subclasses of MPIRT_QOS_TRANSFER
    when naming this type of argument.

  6. S. Hebert discussed Inertia status, work, and issues. He agreed to distribute the
    first version about 6 weeks after the end of the meeting, matching the 3/13/98
    release of the standard draft.

The meeting adjourned at 4:30pm on 2/25/98
The meeting reopened at 8:00am on 2/26/98 [S. Hebert presiding 8-10am, Skjellum
presiding 10am-12n.]

  1. The Instrumentation Chapter was discussed at length. Paavola’s proposal was
    introduced formally to the group. A straw poll was taken 11/0/3 (passed) to
    accept and pursue Paavola’s proposal, and to see it completed.

  2. It was discussed that sets should be contained in homogeneous containers. The
    editors will impose edits to introduce this concept for next time. Examples are
    as follows: probe sets, channel sets, event sets, PGCO sets.

    Skjellum mentioned that it has always been the goal to replace naked arrays and
    counts with homogeneous containers, and that this would improve the object-based
    nature of the API.

    Paavola asked about heterogeneous container objects, which apparently amounted
    to encapsulating all the objects of a single MPI/RT “commit” and did not seem
    to have much purchase with the committee.

  3. Discussion of the level of completeness and timeliness of late-binding issues
    was brought up. Skjellum offered the suggestion that late-binding be postponed
    to MPI/RT 1.1, and suggested a formal vote. This was moved and seconded from
    the floor, and passed 7/0/1. This will have certain impacts in several chapters.

    During the debate concerning this, Skjellum described the general issues associated
    with dynamic support and reconfiguration planned for MPI/RT 1.1, including extensions
    and quality-of-service associated with MPIRT_CHANNELS_TRANSIT (mode changing). This
    met with general approval.

  4. It was requested that there be
    – conditions defined for each object
    – probes that are predefined
    – system event names (POSIX subset) [later, we noted this is 1.1 only]

  5. It was discussed that IMPI chapter would discuss interoperability of MPI/RT’s, not
    interoperability of MPI’s with MPI/RT’s across implementations, nor refer specifically
    to the NIST effort called IMPI.

  6. The parliamentary procedure of approving the minutes at the subsequent meeting was
    requested as point of order. The chair accepted that this would be the practice,
    beginning with the current minutes, to be approved at the next meeting. The chair
    also pointed out that these minutes would be subjected to web scrutiny and revision
    long prior to that meeting.

  7. Thanks were offered by the committee to Dennis Cottel for hosting this MPI/RT Forum

The meeting concluded at 12:15pm on February 26.

Minutes of MPI/RT Forum meeting of January 14-16 in Albuquerque

Ron Brightwell, Sandia, (local host)
Arkady Kanevsky, MITRE, (co-chair)
Zhenqian Cui, MSU,
Steve Paavola, SKY Computers,
Billy Chin, Lockheed Martin, GES,
Anna Rounbehler, Raytheon,
Randall Judd, SSC-SD (USN),
Dennis Cottel, Spawar Systems Center,
Tom Robey, Khoral Research,
Michael Grieco, ASEC (NSA K414),
Anthony Skjellum, MSU, (co-chair)
Nathan Doss, Sanders,
Clay Taylor, MPI Software Tech,
Robert Babb, U. of Denver,
Shane Hebert, MSU,
Robert George, MSU,
Yogi Dandass MSU
Randy Janka Georgia Tech
X. Fowler U. of New Mexico [TO BE CORRECTED ONLINE]

  1. The next scheduled release of the document is January 30, two weeks after
    the end of the January meeting.
  2. Since the December meeting, two additional informal meetings were held…
    One at MSU (R. Brightwell, A. Kanevsky, T. Skjellum, C. Taylor, Z. Cui).
    Second meeting at HICSS’98 in Hawaii (Arkady and Tony).
  3. Rules for voting: One vote per institution.
    Voting institution must have attended at two out of last three meetings.
    There are two readings per each chapter.
    A reading and vote within proximity of each other is preferred.
    The final vote for each chapter of the document at the last meeting.

We consider national laboratories to be separate entities as
compared to their management company’s other activities for purpose
of counting institutions.

  1. Clay Taylor gave an update on Data Reorganization January 13 activities
    -Presentation by A. Rounbehler on Design For Data Reorganization API was given
    -Discussion of data reorganization attributes
    -Discussion of object design for data reorg and relationship to VSIP and MPI/RT.

  2. VIA website is Via has slots for QoS but no
    formal Real Time Model yet on its channels. Interesting possibility.

  3. Status of the Inertia Implementation
    Latest changes to the document have impacted the delivery of inertia.
    Some Questions were raised by Shane:
    What happens if a commit is called. Each time it is called, what resources
    are available to the commit?
    The old Transit operation need to be brought back.
    What happens when inter-communicators are introduced.
    It was decided that there is no problem,
    since relationships occur between communicator groups.

  4. Changes to the document, discussions and readings:
    The January 12 version was posted before the meeting and
    new January 16 version that included many updated chapters was
    posted during the meeting on the web.

-C++ bindings have been partially introduced into document. These
will continue to be added from release to release, and will be discussed
at future meetings from a design perspective.

“Discussion” items now include answers when they are resolved.
They will be turned into permanent material in the chapters or removed
depending upon their significance.

All examples are not up to date. Do not take examples in the text
as representative for now.

Preface - New chapter, TBD for next meeting
Glossary - asked for during the meeting and placeholder chapter was created
in January 16 version. Names of terms and proposed definitions welcome
from all.

Part I - Overview:
A. Introduction for Users - new chapter, TBD for next meeting
Portions of the document “Summary of MPI/RT Basic Communication”
posted on the exploder will be placed in Introduction for Users.
The use of benchmarks for MPIRT. That work is in progress in MITRE, MSU and
Georgia Tech.
B. Introduction for Implementors - new chapter, TBD for next meeting
C. Introduction - new figure for two level middleware was added,
a text to accompany it will be added for next meeting
Reference diagram 4.2 (of Jan 12 version or diagram 3.2 for Jan 16 version)
Need to make changes to mnemonics (L1, L2, L2+). Need
words and descriptions that clarify the levels of middleware.

Part II - Concepts: major revision

A. MPI/RT Objects chapter was moved from Part I into Part II, and
completely rewritten. The chapter was read twice.
AT the first reading January 14 the following agreements were reached:
chapter will be divided into three sections:
Overview of objects, system objects, user objects.
Hierarchy diagram needs more details: needs both system and user objects,
needs to clarify system object classes that have two or one phase
initialization, that allow wait and test operations, that allow data transfer
Requests were discussed. Erequests are not needed. The word request does not
convey useful information. It was agree to create two system object classes:
completable (which support test and wait operations), and transfer (that allows
data transfer operations). For now all objects in class transfer are also
in class completable.

The new diagram for MPI/RT object hierarchy was discussed again January 15.
The two-phased initialization objects will be presented as shaded boxes
rather then separate class. Group, Comm, Datatype are added to the system objects.
A new class was created as a child to system object class: memory descriptor.
It currently has only buffers but in MPI/RT-1.1 scatter/gather buffers
will be added.

Vote to replace old hierarchy diagram with the one presented
pending some analysis by Tony & Arkady. 14/0/2
Vote to change request to completable object [wait/test] 10/1/7
Vote to put buffer back to object 12/0/4
Write up a gather scatter descriptor for MPI/RT 1.1 has been put on hold

The January 16 version contains the new write-up of the chapter
that was read on January 16 (due to its importance to the rest of the
document). Many minor changes were proposed with clarification of the
new object diagram, definition of DUP, and relationship to datatypes being
the most significant.
The new version will appear at the next release.

B. Communicators, Groups and Context chapter - was added.
Currently empty but will contain chapter 5 of MPI-1 with MPIRT prefix,
and with certain other modifications relevant to MPIRT format.

C. Datatypes - chapter will be added to included portion of the basic
datatypes of MPI-1 from chapter 3 plus new datatypes that MPI/RT introduces.
January 16 version has a placeholder for it already.

D. System handlers chapter - major revision from December.
Handler function is fully defined, fanctor introduced. All old discussions
resolved. Quality of service was clarified.
The chapter was read on January 14.
Need to add a clarification on when and how can the handler function can
be changed. Need to add state diagram for handler object.

Chapter revised add read again on January 16.
The failure handlers are gone. They are just handlers that are invoked
on error conditions. Had discussion how to use handlers for monitors.
There was a discussion on light-weight handlers for completion of
data transfer for transfer objects. The name “system handler” will be
changed to “handler”.
Need clarification on multiple handlers (may be from different libraries)
on the same condition of the same object. The order of handler schedule
is determined by QoS.

Need a few more operations to easy a search
for multiple handlers on the same condition, with the same handler function
and so on. Steve Paavola agreed to do that.

Introduce the object name for non channel objects. This will allow
implementors simple rules for matching objects.

E. Buffer management chapters - minor revision:
New diagram for buffer management and old discussions resolved.
No reading on this chapter was done only some discussion.
The new subsection for data descriptors with buffer objects will
be added for the next document.

F. Channels chapter - No major revisions:
No discussions or reading were done on this chapter.

G. Collective operation chapter has been completely rewritten:
Broadcast, Gather, Scatter, Reduce and Barrier, have been included.
Some of the operations needs two set of bufiters one for sending
and one for receiving.

This chapter was discussion on January 15.
Barrier needs to be converted into MPI/RT format with the collective channel
with QoS. Each of the collective operations needs its own set of
get/set, dup, free operations, not just create.

Need full details on how to implement a collective operation as a collection
of point to point channels. This potentially include the creation of
new bufiters, bufiter of different sizes over the same memory and sharing
bufiters between point-to-point channels that are created by implementors.
This is basically an advice to the implementors how collective operations
can be layered on top of point-to-point operations.

The following were also noted during a discussion;
The differences in the number of buffer iterators for the different collective
operations (broadcast, gather, reduce).
Some parameters make sense only for the root (as in MPI-1).
-Discussion on what additional attributes are needed in buffer iterators
to support collective communication. The conclusion is that none are required.
Handlers were discussed with respect to supporting collective communications.

H. Data Transfer chapter - is completely rewritten.
Discussed on January 14 together with MPI/RT objects chapter.
Rewritten in January 16 version to be in synchrony with MPI/RT object chapter.
MPIRT_DO is gone; Wait and Test is on completable objects, all
are on transfer objects. The transfer objects are all completable.

Part III - Models, had not been updated.
It is out of synch with. the rest of the document.
The changes for MPI/RT object chapter has major impact on it.
It will be updated for the next version of the document.

Part IV - Application management: major revisions:

A. MPI/RT Initialization and Termination has been written.
It was not included in the January 12 version, it is in January 16
version. It was not discussed.

B. Clock Chapter have not changed over the last year.
Chapter was read on January 15.
The following changes were requested:
MPI_WTIME to be changed to MPIRT_WTIME.
Change globally synchronized to master clock.
Define which clock (master clock or local).
Clarify the use of local, synchronized, global clocks when they are used
in light of new master clock and definition of synchronized clock which is

Add to the text, MPI/RT is required to use a synchronized clock.
Advice to implementers should state recalculate not reset the drift parameter.
Define constant MPIRT_WTIME_UNDEFINED and use it for skew,
accuracy and access time and all other places where implementation can not
provide a bound. That takes care of the caveat and discussion at the beginning
of the chapter.
Add advice to implementers to clarify that the platform
or the MPIRT services can support the MPI/RT master clock.
The definition for MPIRT_TIME_OBJECT will be moved to datatype chapter.
Need a definition for Epoch. Recommend using implementation defined
MPIRT epoch with extra function to find the offset from
the POSIX definition of epoch.

The chapter was updated for January 16 version and read one more time on
January 16.
Many minor word-smithing exercises were done.
The new datatype MPIRT_DATATYPE_LONGLONG will be introduced in datatype
chapter, which new datatype MPIRT_DATATYPE_TIME is using.
This is needed to separate time from dimensionless drift which uses
the same size datatype.
All measurements are in nanoseconds.
to the places in the chapter where they are defined.

The chapter was VOTED on. 11/0/0 (institutional [formal] votes).

C Instrumentation chapter - has been updated.
It was not included in the January 12 version, it is in January 16
version. It was discussed on January 15.

It was agreed to support one API for instrumentation and monitoring.
This API would provide a forum approved selection of instruments
for MPI/RT and have a design that allowed expansion in the
future to add other instruments.
This additional functionality is deferred to MPI-RT-1.1.
To avoid unnecessary over head, a wait and test object for monitors
has not been included. The ephemeral (erequest) has been removed.
The monitor can
use the handlers instead of the ephemeral request.

Part V - Non-Real-Time Services - no changes.

Part VI - Interoperability with other standards: major changes: No discussions.

A. MPI: a small subsection on MPI_initialize
and finalize was written in January 16 version. Still incomplete.
Need a discussion in the document that describes the progress guarantees
for MPI with MPI/RT.

B. IMPI: empty - Thom McMahon and SHane Hebert will contribute text
for February time frame.

C. VSIP: preliminary version appears in January 16 version. Puri
Bangalore and Randy Janka will meet to discuss before next meeting.

D. Data reorganization: empty - Clay Taylor will contribute text
by February meeting.

E. Packetway: empty. Robert George will provide text by February meeting.

F. VIA: new chapter. Preliminary version appears in January 16 version;
Rossen DImitrov will offer additions after he gets feedback.

G. COE: preliminary version appears in January 16 version.

Part VIII: MPI/RT-1.1 - new part that include parts that where moved from
old document per agreement at December '97 meeting.

Meeting adjourned at 2:30pm Friday January 16.

MPI/RT Forum Meeting Minutes December 11-12 2021 San Diego, CA

Ron Brightwell, Sandia,
Arkady Kanevsky, MITRE, (co-chair)
Zhenqian Cui, Mississippi State University,
Steve Paavola, Sky,
Chris Hayden, Sanders (a Lockheed Martin Company),
Darwin Ammala, MPI Software Technology, Inc,
Billy Chin, Lockheed Martin, GES,
Richard Steinberger, Lockheed Martin, GES,
Anna Rounbehler, Raytheon,
Randall Judd, SSC-SD (USN),
Tim Singleton, Navy Peo USW,
Dennis Cottel, Spawar Systems Center, San Diego,
– local host
Keith Bromley, Spawar Systems Center, San Diego,
Tom Robey, Khoral Rsearch,
Michael Grieco, ASEC (NSA K414),
John Williams TASC (NSA K414),
Anthony Skjellum, Mississippi State University, (co-chair)

December 11,2021
I. Overview

  1. Review of issues left over from previous meeting:
    Discussion of Activate/Deactivate

  2. Email comments that have not been incorporated

  3. Status on Inertia, the MPI/RT prototype - Most of code is untested.
    Buffer Management
    Buffer sets 90% written
    Buffer Iterator 90% written
    Buffer Iterator Accessor 80%written

    Handlers 80% written
    Channels 60% written
    Channel set 50% written

    Collective Operations 0%
    Data Transfer 50%
    Quality of Service 80%

    Time driven
    Event driven
    Priority Driven

    Need to have more questions answered by Reflector
    Linux DARPA work will be the basis of Action prototype
    Maruti (time based scheduler) will be available
    Need VXworks to be modified - use third party add-on.

  4. Comments on Inertia by T. Singleton
    Money is limited to a Fiscal year
    Vendors feel that it is langishing
    No coordination with VSIP
    Responsibility of MPI/RT to integrate to VSIP
    Recommendations - lock down and baseline the spec.
    New schedule for release of Inertia.
    Concern expressed for no formal votes taken
    Need to show solid progress
    Need plan to solve the VSIP - MPI/RT compatibility problem
    Problems with copies of special buffers
    Export Paradigm proposed. Buffers are visible to the user.
    User can perform computation on channel buffers.-VSIP.
    VSIP memory is not visible to the user

  5. Comments by Tony -
    Issues to resolve
    Implementation of Admission tests will need more work
    Advice to Implementers need to be added to the document
    Goal is to finish the standard by the end of June
    Other related efforts
    DARPA funded efforts to investigate OS and real-time schedulers
    DARPA effort to implement Maruti
    Windows C2E for real-time
    Need improved scheduler for various OS to support MPI/RT -
    third party product with real time extensions

  6. DII-COE is certifying Lynx as standard OS.

  7. Tony and Arkady are developing a 3-day Tutorial on MPI/RT. Several hundred slides.

  8. Need a Web Site with information that describes what is MPI/RT.

  9. Arkady & Tony will present a “schedule” for products to be developed.

  10. Desire to go to ANSI standard for acceptance upon completion of the standard.
    Political problems of bringing MPI-1.x functionality into ANSI.

II. Discussion of MPI/RT:

  1. Request and Object Handles.
    A. Object types: user and system objects
    Object contains: Object description, handle, user invisible physical object,
    and potential implementation defined attributes.

Common System objects operations: Create, Free, Dup, Get, Set.

Commit operation builds physical object from its desription.

B. Requests
Agree with Tom’s email comment about nomenclature should be requests.
Hierarchy suggestion STRAW VOTE - passed
Channel Collective channel Channel sets

Should we have unique START, WAIT , TEST, PROBE in MPI/RT?
TEST and PROBE are costly.
CANCEL is rejected.
Mercury has requested a polling technique - poll on completion - TEST.
Need a specially defined handler for this functionality.
Proposal for functionality for data transfer operations
Straw Vote passed.
START is fire and forget about it. DO is fire and the ability to monitor.
There are two events on a channel that are of interest (atomic operation - data
transfer completed end to end) and (local completion
where the local request was satisfied).
How does the concept of TEST fit into the concept of performance monitoring? - The user
would like to take action based on monitored conditions.

  1. Should MPI-1.0 calls be mixed with MPI/RT?

  2. The chapter on collective operations is very outdated and needs work.

III Discussion of Collective Operations, Channels and Requests
On hold

IV Handlers - Section 7.1

  1. History
    Desire to have more than a single handler for channel objects.
    Local and global completion of data transfer operations conditions.
  2. Start of Data Transfer needs to be added.
  3. Need to define MPIRT_handler_fn arguments
    What are the arguments? (object , communicator, handler functor)
    Handler functor(extra state, condition and other parameters from handler create).
  4. Handlers can have handlers which have handles (cascading handlers).
    Example with failure function. Discussion on two level hierarchy.
    Failure_handlers do not allowed to have failure handlers.
  5. Handlers can be used for other objects:
    See page 8 for list of objects - there will be handlers for all except bufset.
    STRAW VOTE Passed
  6. Need clarification of Commit Rules
  7. Need clarification of DUP rules
  8. Proposal: To replace extra state by attribute caching - problem
    with embedded architectures.
    Refer to MPI-1.0 section on attribute caching.
  9. Proposal: Single attrribute (ptr, length, copy callback, delete callback)
  10. The implementation must treat a DUP channel as if it were created using Channel_Create.
  11. Meaning of Uncommitted, Committed and Dup are the same as for all MPI Objects:
    Straw Vote 10 yes, 3 abstain

V Other Suggestions
Assign tasks to various members of the forum to write examples

December 12, 2021

I .Review of Discussions -
COMMIT can be undone with a FREE.
DUP can be done only on a channel that is not COMMITTED.
Undecided as to whether committed channels can be DUPed.

II. Logistics
Proposed Schedule (the complete schedule will be posted on the web
and on the reflector)
First Readings Other
January,2022 Chapter 2 becomes chapter 5
Chapter 1,2,3,4,6 Chapter 5 becomes 16

February 2022
Chapters 7,9,10,15 Chapter 7 System Handlers will be moved
19-22, 16

April, 2022
8, 11,12,13, Part VI (23-28)
Tutorials (4 planned)

    Remove chapters 18
    Designate 14, 16, 17 for MPI/RT II
    Add 2 Chapters (1 and 2)
      Advice/Tutorial Introduction

Future Meetings - will be extended to three days
The status of a chapter will be in the first few lines of the chapter
Chapter 8 - collective operations (broadcast, scatter, gather, reduce, barrier)
Reshape in MPI/RT2
Chapter 16 Instrumentation deferred to MPI/RT 2.0. Do we want to do this?

III Straw Votes
Proposed Schedule - Unanimous
Move Chapter 2 to Part II 14/0/2
Move chapter 5 to Part IV 15/0/1
Split chapter 7 14/0/2
Chapter 8 -Reshape deferred
(bcast,scatter,gather,reduce,barrier) 14/0/2
Chapter 14 deferred - see 1.1 16/0/0
Chapter 16 Instrumentation deferred 3/6/6
Chapter 16 First Reading Feb. 12/1/3
Remove Chapter 17 16/0/0
Delete Chapter 18 14/0/2
Reorg VI and add layering MPI-1 chap 14/0/2

Need to address: How does instrumentation monitoring “fit”
into the PMPI monitoring concept?
Keith Bromley suggested we take advantage of Fault Tolerance and Handling
(Sanders, DARPA, JPL) efforts

IV Iterators
Recall that System Handler and Event Handlers differ.
Event handlers are more permissive.
System handlers are local while event handlers can be local and global.
Problems with System handlers: QOS, re-entrant problems
Intent of System Handlers - instrumentation, scheduling system resources
(this handler has x time units to run),
used with other objects (see page 32)
Proposal to change QOS for system handlers
Straw Vote -Proposal to make handlers functions re-entrant, thread safe,
full weight -advice to users. PASSED
Proposal to change arguments in QOS - agreed (no vote needed)
Page 34 - Shane pointed out that extra state has variable length arguments
which is a problem.
Will be re-written as part of the pervious vote.
What happens when there is an error on an object, if the handler function is called?
How do you recover from an error condition when QOS is not met?
How do you restart a channel?
How do you specify a recovery policy and access
the buf-iterators for control of the buffers?

V MPI/RT Initialization/Termination
A proposal for Chapter 13 Initialization is made by Tom (Khoral Research)
This seems out of the scope of MPI/RT.

A layered approached for MPI-1 and MPI-2 on top of MPI/RT discussed.
Need MPIRT_Init, MPIRT_Finalize, MPIRT_Finalized, and MPIRT_Initilialized.
Straw vote taken, passed unanimously.

Minutes of the MPI/RT Forum on November 6-7 in Boston


Randall Judd SPAWAR Systems Center
Dennis Cottel SPAWAR Systems Center
Michael Grieco ASEC - NSA N41
Thomas Robey Khoral Research
Anna Rounbehler Raytheon
Arkady Kanevsky MITRE (co-chair and host)
Jerrell Watts Syracuse U.
Shane Hebert MSU shane@ERC.MsState.EDU
Zhenqian Cui MSU
Manoj Apte MSU manoj@ERC.MsState.EDU
Tom Daggett NUWC
Robert Cunningham MIT Lincoln Lab
Richard Games MITRE
Chris Hayden Sanders, Lockheed Martin Co.
Billy K. Chin Lockheed Martin, GES
Richard Steinberger Lockheed Martin, GES
Leonard Monk CSPI
Clay Taylor MPI Software Technology
Robert Babb U. of Denver

The following topics were discussed:

  1. Dennis gave a short briefing on Data Reorganization meeting that
    preceded MPI/RT Forum.
    (see coming soon).
    For the future, we will coordinate MPI/RT meetings
    with Data Reorganization meetings.

  2. Shane presented the current status of IneRtia.
    The discussion started on the progress engines in MPI, MPI/RT and
    IneRTia implementation. It gradually spread on the issues of
    thread and signal safety, enabling and disabling interrupts
    at a user specified time… At the end it was recommended to Shane
    to make appropriate decision for a progress engine for IneRTia
    and document it properly; it may require an extra operation,
    which is unique to the IneRtia, that users call to progress

  3. There was a prolonged discussion on objects in MPI/RT and their
    potential interaction with MPI “objects”.
    The following straw votes were taken:
    “A channel is a kind of Request and can be used everywhere Request is”
    (accepted 11/0/3);
    “Get rid of MPIRT_Channel_get_request call”
    (accepted 10/0/3);
    “Add distinction between objects and handles
    (objects that have handles (system objects), and other objects - opaque
    (user objects))”
    (accepted 15/0/0).

    The last vote amounts to creating object hierarchy.
    There is a need for up front chapter to describe it,
    a need to clean the existing bindings and an email on
    reflector to propose the exact terminology. (This was assigned to
    Tony and Arkady.)

    There was a discussion on committed and uncommitted “system” objects.
    It was agreed that errors and clarifications will be added for
    appropriate operations.

    It was agreed that object FREE operations return NULL handle.

  4. There was a discussion on buffer iterators and their functionality.
    The main topics were blocking vs. nonblocking behavior, overflow and
    underflow behavior, and insert/remove operations.

    The following straw vote was taken:
    “No error for no buffer available for MPIRT_Start & MPIRT_Wait
    for either channel end should be raised. The QoS error may be raised”
    (Accepted 10/0/3).

    There was a discussion to associate “error/blocking” policy on
    buffer iterator or channel.
    No conclusion was reached, the issue will be revisited in San Diego.

    It was agreed to add MPIRT_BUFITER_COUNT_BUFFERS(bufiter, bufcount)
    that returns the number of buffers in the buffer iterator.

    Several minor changes, like positive vs. non-negative for various counts,
    were agreed on.

    It was agreed that buffer addresses are not unique and the same
    buffer address can be used for multiple bufsets to support
    data fusion. The appropriate text and examples will be added.

  5. The data transfer operations were discussed.
    The following straw votes passed:
    “The document will contain three distinct functionalities:
    Start and Wait (MPI-1 like),
    MPIRT_DO (start without wait),
    Activate and deactivate (user will not start or “do” at this end of
    a channel, the other endpoint of the channel, time calendar, predefined
    events and so on will start the channel data transfer)”
    (Accepted 13/1/2);
    “Allow multiple channels to be active when they share iterator”
    (Accepted 12/0/3);
    “The QoS specified on two ends of the channel can be different”
    (Accepted 13/0/3).

  6. There was a discussion on performance for MPI-1 type functionality vs.
    degrading performance (tightness of achievable schedules) and predictability
    for RT functionality.

  7. There was a request to add index to the document and a section on
    terminology and definitions.

  8. It was agreed that we will switch to the three day format starting
    from the Albuquerque meeting and until there is a need for it.
    The Albuquerque meeting will start a day earlier on Jan 14.

  9. There was a discussion on the Momentum (the first real-time supporting
    implementation prototype). The main topic was what to include in it.
    The following areas were discussed: preemptive scheduler (at time …),
    preemptive handlers, clocks and timers, profiles and instrumentation,
    and an ability to pass to implementation
    allocated resources and schedules.

The meeting adjourned at 3pm. on Friday, November 7.

Minutes of the MPI/RT Forum on September 24-26 at Mississippi State University, Starkville, MS


Jerrell Watts Syracuse U.*
Ron Brightwell Sandia
Michael Grieco TASC*
Thomas Robey Khoral Research*
Dennis Cottel NRaD*
Arkady Kanevsky MITRE (co-chair)*
Yogi Dandass MSU
Robert George MSU
Shane Hebert MSU
Greg Henley MSU
Anthony Skjellum MSU (co-chair and host)*
Zhenqian Cui MSU
Peter Brennan MPI Software Technology*
Darwin Ammala MPI Software Technology
[* = institutions present that are able to vote in formal votes at
this meeting.]

The following topics were discussed:

  1. The document has been reorganized as a standalone standard,
    as has been discussed on several occasions. The outline was
    reviewed, and further structural changes were made between
    Wednesday and Friday, including the introduction of “parts”
    to the document in order to group sets of chapters. Several
    new placeholder chapters were introduced to help firm up the
    scope of what the final document is to cover.

A number of changes discussed below have been effected during the
meeting, and an interim release of the document is anticipated early
next week.

  1. The Inertia prototype was discussed (Shane Hebert presented).
    The problems with buffers, channels, and templates were examined,
    with recommendations to the committee for correction.

  2. Templates were discussed, and it was voted to remove templates
    in favor of new, but upward compatible technology from Jerrell
    Watt’s recent write-up (distributed at the meeting). Shane had
    also expressed desire prior to meeting to get rid of templates. These are
    object descriptors (which are part of the opaque objects).

There will be a one-to-one correspondence of object descriptors and
their underlying real objects. Straw vote: 9/0/0. As a consequence,
channel and channel_set objects cannot be committed twice.

An object descriptor’s parameters cannot be changed once the object
is committed.

Discussions of “dup” (shallow vs. deep copy) ensued. Shallow copy
will be supported. [Straw vote on shallow vs. deep: shallow
accepted by 6/0/3.]

Discussion on adding a “deep dup” ensued. Failed by straw vote of 1/4/3.

  1. Closure on the issue of buffers, iterators (replacing bufqueues),
    and buffersets led to the clean model we have been seeking over
    last few months.

Buffers will have ranks in buffersets, as well as labels.

  1. Handlers were discussed. We voted to merge handler & error handler
    classes. Straw vote 6/0/1. We decided that error handlers are of type
    “system handler” see below, discussion item #14).

  2. Prolonged discussion on buffers, buffersets, iterators, etc, was
    undertaken. We discussed two iterators per channel, labeling of the
    buffers (by integers), and indexing of the buffers by iterators.
    The following straw votes were taken:
    a. memory allowed to be used for multiple buffers: 8/0/3 in favor
    (advice to users, advice to implementors will describe this)
    b. we decided that two new statuses (stati?) or events will be
    added: “when user asks for a buffer and the iterator is empty”
    and “when buffer is returned to the iterator and the iterator
    is full”
    voted (unanimously): 11/0/0

    This defines non-blocking behavior.

  3. Channels were discussed, and new operations to activate or deactivate
    channels or sets of channels were proposed. These are, in essence,
    to start or stop schedules that were created by admission criteria.
    [Supports mode changes.]

Straw vote was taken to remove old, non-template version of the
channel semantics. Carried: 8/0/0.

  1. Quality of service was discussed. If the periods are defined for
    the time-driven model, then relative start and deadline are defined
    with respect to the start of the period, not the time at which the
    operation was posted. Otherwise, start events or start have be an
    absolute time or an event, and the deadline, if relative, is to be
    with respect to the start. We allow deadlines to be longer than
    the period, with appropriate advice added to the document, defining
    issues concerning quality of implementation.

  2. Jerrell/Tony will meet soon (September 27) to make further updates
    to the document for the new style of collective channels that have
    been agreed on, including exploitation of the iterators. This will
    lead to support for the MPI-type collective operations in the face of
    MPI/RT channels. This discussion will include review of the basic
    API discussed under item 6.

  3. The recent proposal of Anna Rounbehler/John Lebak was discussed,
    and people promise to give feedback. The description of the buffers
    and how the data is distributed across the communicator were missing.
    The indices currently present are seen to be insufficient. The mapping
    to the actual memory where data resides locally is unclear.

In addition to the needed changes, the updates reflecting our new gestalt
with collective operations will have to be incorporated.

  1. Relationship to other standards (COEs, TASP, VSIP, IMPI, DARPA Data
    reorganization) were covered.

  2. We discussed the issue of MPI_, MPIRT_, and MPIFT_ headers, for
    the base, RT, and fault-handling technology. We agreed to stay with
    the various headers, as these make the most sense from the point of
    view of portability.

  3. We discussed a single handle for a channel_set, and decided to have
    this. The main feature is that the request that represents the channel
    set shall be analogous to a channel request. Semantically identical
    to collective channel.

  4. We agreed on the lower-level event paradigm is no longer in the
    real-time paradigm, but fits into the “Concept” part of the new document.
    There are two types of handlers: system handlers, and event handlers.

  5. Profiles are behind schedule, and really will be in place for
    November meeting.

  6. Logistical issues:
    next meeting at MITRE, November 6-7; coordination with other
    meetings underway

    following meeting at San Diego (NRaD or hotel); coordination
    with VSIP has been done, other coordination under way

  7. The web page is now active. The associated majordomo
    is setup, and a copy of existing Argonne list members will be added,
    informed about the new list, and given the option to delete themselves.

    Pointers from the previous web site,, point
    at the new web site.

  8. It was announced that Yogi Dandass will be responsible for document and
    web page updates (, 601-325-4601), and will
    assist the chairs.

The meeting adjourned at 2:50pm on Friday, September 26.

Faithfully submitted,
Tony Skjellum & Arkady Kanevsky

Minutes of the MPI/RT Forum on July 24-25 in Boston


Joe Germann SKY Computers
David Schwartz Hughes
Gerard Vichniac Mercury Computers
Tim Singleton NAVY
Jon Hiller Science & Technology Associates
John Williams TASC/K41 (NSA)
Donald Shakley MITRE
Shane Hebert MPI Software Technology Inc.
Randall Judd NRaD
Thomas Robey Khoral Research
Anna Rounbehler Raytheon
Arkady Kanevsky MITRE (co-chair and host)
Anthony Skjellum MSU (co-chair)
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Jim Ionata NUWC
Robert Cunningham Lincoln Labs
Bob Ford MIT Lincoln Labs
John Ramsdell MITRE
Paul Husted MITRE
Chris Hayden Sanders
Nathan Doss Sanders
Mike Butler MITRE
Richard Games MITRE
Stephen Rhodes Advanced System Architectures
Billy Chin Lockheed Martin, GES

The following topics were discussed:

  1. Tony gave an overview of the current status of MPI/RT Forum.
    The highlight of that was a schedule with deadlines for various sections
    of the document. As usual the talk will appear on our web
    ( soon.

    The schedule for chapter completion will be put on the web site
    prior to the next meeting as well.

  2. For the future, we will coordinate MPI/RT development and meetings
    with VSIP and TASP OS development and meetings.

  3. Templates were discussed in details.

    The view of the templates from the OO class point of view will be added.
    The templates for individual object then become derived subclass,
    which share the same set of operations.
    We voted to delete “modify” and “create empty templates” through out
    the document.
    We voted to add “free” operations that allow to delete a template
    instance through out the document.
    We discussed operations that allow application to set up the default
    values for template classes and derived classes. The pro and cons
    of this preproposal were discussed, with the majority opinion that
    users already have this functionality using “dup” and “set” operations.
    The full proposal on that can still be entertain at the next meeting.

    We discussed the possibility of reducing the number of “set” and
    “get” operations currently in the document by having a single “set”
    and “get” operation with one extra input parameter which specifies
    which template parameter to modify. At the end the group agreed to
    keep the existing API for clarity of use.

    We agreed to change the names of the template operations to keep
    “template” and template object name together and move the action
    to the end of the operation name (e.g., MPIRT_TEMPLATE_BUFPOOL_CREATE).

    The name bufpool will be reconsidered with potential change to
    BUFFERSET. This may trigger further changes in the operation names
    trying to achieve consistency in the naming convention.

  4. The binding issue was partially discussed concentrating on delaying
    FORTRAN binding, and potentially adding JAVA binding in the future.

  5. The clock attributes were discussed again. The group voted to
    make current attributes read-only. The scope of the synchronized clocks
    was discussed, old idea of having attributes per communicator was
    reintroduced, as well as when the clocks should actually be
    synchronized. The proposal on this topic will be considered at the
    next meeting, but may not be part of the December '97 MPI/RT complete

  6. The profiles were discussed in great length.
    We voted to add subset of MPI-2.0 instead of full MPI-2.0 functionality.

    At the end the following 4 profiles were voted in:

    • Resource constrained MPI-1.2 and MPI/RT,
    • MPI-1.2 and MPI/RT,
    • Resource constrained MPI-1,2, subset of MPI-2.0, and MPI/RT, and
    • MPI-1,2, subset of MPI-2.0, and MPI/RT.

    The resource constrained MPI-1.2 in itself is not a real-time profile
    of MPI.

    The subset of MPI-2.0 to be supported will be determined in future
    discussion. It is anticipated to emphasize dynamic process management.

    • There were multiple requests for including a list of functions as part
      of the definition of
    • resource constrained MPI-1.2 systems.
    • MPI-2.0 subset.
    • It was suggested and agreed upon by consenus to give each profile a
      specific name.
  7. Discussion of buffer pools (now officially called BufferSets) was
    lively. It was determined after significant discussion that clarification
    of the behavior under store-and-forward is needed. It was decided
    that there may be some issues regarding “implicit” mode. It was
    argued that the BufferQueue objects need to have a well-defined
    initial state, and that clear functionality for both the get and
    put side must appear in the standard.

  8. An overview of PacketWay was given by Tony in regards to providing
    interoperability services for several MPI or MPI/RT implementations.
    [It was again commented that interoperability is part of the
    MPI/RT effort.]

  9. It was discussed to add the ability for handlers on send and receive
    side to inherit the priority of the channel they are transmitting on.
    [This makes sense definitely for receive side, send side TBD.]

    Issues concerning the QoS of handlers
    * time to begin execution
    * duration of execution OR priority of execution
    was given, commenting that these two issues are orthogonal. The
    ability to guarantee a time bound for the commencement of execution
    was debated.

  10. Discussion of restructuring of the document into chapters occurred;
    it is agreed that for September, the fully restructured document
    would be offered. It would be offered actually prior to next meeting,
    by the deadline of August 27. This will give significant time for
    pre-meeting discussion on the reflector.

  11. The need for an extra day of meeting (2.5 days) starting with September
    MSU meeting was discussed and accepted. The meeting at MSU will be
    so scheduled.

  12. It is noted that consideration of VSIP’s buffering approaches
    should be undertaken (cf,

    VSIP attendee Dave Schwartz noted that future processors may not
    offer single address spaces, and that VSIP explicitly did not
    make such special memory available to users of VSIP (e.g., SRAM pages).

  13. We intend to put online resources on the MPI/RT Web page, related
    to: TASP, VSIP activities. [Tony to do this].

    Vote to remove template modify operation 10/0/1 carries
    Vote to remove create_empty operation 10/1/0 carries
    Request for class-based initialization of templates 2/6/10 fails
    Add template free operation 14/0/4 carries
    Make Clock parameters (section 8.5.4) read-only 11/0/6 carries
    Define 4 profiles (see above) 15/1/2 carries
    (Resource Constrained MPI 1.2 is not MPI/RT!)

The meeting adjourned at 12 noon on Friday, July 25.

Minutes of the MPI/RT Forum on June 5-6 in Boston


Dennis Cottel NRaD
Thomas Robey Khoral Research
Anna Rounbehler Raytheon
Arkady Kanevsky MITRE (co-chair and host)
Anthony Skjellum MSU (co-chair)
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Jim Ionata NUWC
James Lebak MIT Lincoln Labs
Rob Steele MIT Lincoln Labs
Daya Atapattu Scientific Computing Associates
John Ramsdell MITRE

The following topics were discussed:

  1. MSU had established the web page for MPI/RT.

    The JOD version of our chapter is there in .ps and .pdf formats.
    All past and future drafts will be posted there in .ps and .pdf formats.
    Future meetings schedule is there as well as other MPI/RT related info.

  2. We will investigate the possibility of rescheduling MPI/RT meeting
    to be back-to-back with VSIP and TASP OS meetings. More on that
    via email, and on the web site under “Meetings”.

  3. The channel proposal was discussed.
    We agreed on a layered template approach with user incrementally building
    channel and channel set information with a single commit operation
    for a channel set over a communicator. No actual objects like channel
    requests, channel set handles and others do not exist until the commit
    operation is performed. Only templates and handles to them exist
    prior to that. This allows a very clean layer library design as well as
    object hierarchy, while actually enhancing the optimizations possible
    by MPI_Channels_Init et al.

  4. The buffer management was revisited again.
    Following the above layered approach the templates were created for
    buffer pools and queues that are passed into channel templates.

    According to the revisions made, users are allowed to create their
    own buffers and pass addresses into a template. Users can also
    request for a system to create buffers that will not be created
    till the channel set commit is returned. Analogously, for queues,
    users can provide persistent generalized requests (user defined
    functions) for queue operation, or specify one of the MPI/RT
    provided ones. The queue handles will not be return till the
    channel set commit is returned.

    The buffer access operations will be modified, and a new “true size”
    operations will be introduced.

  5. As part of the generalized request discussion related to the queue
    management, “start” events will be reintroduced for lower and application
    level event-driven paradigms.

  6. The profiles and the relationships between MPI/RT, JOD, MPI-1.2 were
    discussed and the new figure presented. This discussion will be
    added to the next draft and the figure will be cleaned up.

  7. We agreed that there is only one class of priorities, hence priority
    class bullet for QoS for priorities will be removed. We voted in that
    preemption quantum can be specified in time units or message length

    We also voted to merge priority paradigm with event-driven one. This
    allows to specify priority as QoS for event handlers and data transfers.

  8. We agreed to investigate full-weight and light-weight handlers.
    The light-weight handlers will not use call back mechanism but will use

  9. For application level event-driven paradigm we will investigate
    receiver based model with “labeled/tagged” messages.

  10. Collective operations were discussed. We agreed to pursue channel based
    collective operations with QoS based on collective channel idea.
    The transition of MPI-1 and MPI-2 operations with timeouts will be put
    on the back burner in favor of real-time functionality and reshape
    operation(s) which will appear in the document for the next meeting.

  11. The MPI/RT draft still is not transitioned into stand-alone document.
    This will be done prior to next meeting. Appropriate chapter divisions
    for the first cut at the stand-alone document were discussed (again).

The meeting adjourned at 11am on Friday, June 6.

Minutes of the MPI/RT working group on April 25-26 in Chicago


Dennis Cottel NRaD
Thomas Robey Khoral Research
Anna Rounbehler SKY Computers
Arkady Kanevsky MITRE
Ron Brightwell Sandia National Labs
Robert Babb U. of Denver
Anthony Skjellum MSU
Harish Nag Intel,SSPD
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Peter Brennan MPI Software Technology Inc.
Tarek Abdelzaher U. of Michigan

The following topics were discussed:

  1. Gordon High Performance Conference was announced.

  2. Tony will explore copyright transfer from U. of Tennessee
    to MSU for future MPI/RT drafts and MPI documents.

  3. A long discussion on the simulators of QoS for various platforms
    how to achieve it, how to use it, how to optimize it,
    was conducted. This has a direct affect on implementation strategies.

  4. Multiple levels of implementation activities were proposed:
    API prototype with empty QoS guarantees, a generic QoS guarantee
    for a neutral platform, and optimized QoS guarantee for real-time

  5. The channel section was discussed.
    A proposal was presented and discussed on creating objects to
    reduce the number of parameters for channel management operations
    and simplify their manipulations. The six page proposal was
    written and distributed at the meeting and will be discussed at
    the next meeting.

    A split of collective channel operations into two stage process:
    reserve and commit was proposed. A joint operation which
    allows all channels (collective and point-to-point)
    to get a complete picture on all needed resources collectively
    over a communicator was also discussed.
    The above mentioned object proposal supports this concept.

  6. Several topics on Buffers were discussed. Buffers addresses as
    IN/OUT parameters proposal was reintroduced. The vote was passed to pursue
    the following model:
    Nondestructive buffer send, with explicit wait and freeing the buffer.
    All buffer operations are explicit (including all real-time data
    transfer operations. Buffer iterators specified via strategy.
    The other issues discussed were: who owns the buffers
    application/implementation, buffers replenishment for buffer pools -
    buffer passing between application and implementation, sharing buffer
    pools between channels, buffer pool strategies and their specifications.

  7. A discussion on mismatches of QoS for two endpoints of the
    point-to-point channels, and multiple endpoints of collective
    channels was conducted.
    For the client-server paradigm only one end specifies QoS and the
    other end accepts/rejects it. An analogous situation exists
    for the root of a collective operation.

  8. The current MPI/RT chapter will become a stand-alone document.
    There was a discussion on the structure of the new document.
    The new chapters will follow the logical breakdown of the
    sections in the current document. The exact mapping of sections
    to chapters will be discussed at the next meeting.
    Additional chapters could include: introduction, dependencies on
    MPI-1,2, collective operations, objects/handlers/request and other
    underlying principles, threads.
    Fault-tolerance will be renamed to fault handling.

  9. Future of the MPI/RT was discussed.
    We agreed to continue meetings on the 6 week basis.
    Here is the tentative schedule of the future meetings:

Dates Place Local Contact
June 5-6 MITRE, Boston Arkady Kanevsky,
August 7-8 MITRE, Boston Arkady Kanevsky,
September 25-26 MSU Anthony Skjellum,
November 6-7 MITRE, Boston Arkady Kanevsky,
December 11-12 NRaD, San Diego Dennis Cottel,
January 15-16 Sandia, Albuquerque Ron Brightwell,
February 26-27 NRaD, San Diego Dennis Cottel,
April 9-10 U. of Denver, Robert Babb,
May 21-22 INTEL, Portland Harish Nag,
June 29-30 MITRE, Boston Arkady Kanevsky.
All meeting are 2 days: Full day first day and half day the second.
    The BOF meetings or tutorials will probably be held at
    SC'97  (High Performance Networking and Computing Conference) and 
    RTSS'97 (Real-Time Systems Symposium) conferences.
    We agreed to change our meeting name to MPI/RT Forum.
    A more rigorous voting and appeal rules will be introduced next time.
    The JOD will be published at the same time as the MPI-2 standard document
    as a stand-alone document that contains a chapter on MPI/RT.
    The current version will be presented there with the following changes:
    A. The first figure in the JOD MPI/RT chapter will be replaced by the
       current figure in the stand-alone chapter.
    B. Delete C++ binding (The group voted on it unanimously).
    C. Add disclaimer and the pointer to the ftp/html site of MPI/RT.
    D. Revert Buffer Management section to the previous version.
       Add the state diagram to it.
    E. Fix Appendices. Only the second appendix should be present that
       will be numbered A.
    F. Change Fault-Tolerance to Fault-Handling (voted in unanimously for
       JOD and all future versions).
    G. Put the two new figures from the buffer section that describe the
       sequence of application commands to the profile section.
    F. Add a figure to the profiles sections that illustrates relationships
       between, MPI-1, MPI-2, and MPI/RT (Venn Diagram).


MPI/RT Standard Drafts

MPI/RT 1.1 Standard Draft (July 1, 2024 version)
Postscript Acrobat(PDF)
MPI/RT 1.1 Standard Draft (March 26, 2024 version)
Postscript Acrobat(PDF)
MPI/RT 1.1 Standard Draft (February 26, 2024 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 10, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 12, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (October 13, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (September 24, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (September 8, 2023 version)
MPI/RT Standard Draft (June 7, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (March 5, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 1, 2023 version)
MPI/RT Standard Draft (December 2, 2022 version)
MPI/RT Standard Public Comment Draft (September 21, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (July 20, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (July 1, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (June 22, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Engineering Practices Draft (May 18, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (May 18, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (May 3, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (April 7, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 23, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 15, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 16, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 12, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 31, 2021 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 11, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (November 25, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (November 3, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (October 3, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Draft (July 9, 2021 version)
Postscript Acrobat(PDF)
Realtime JOD Version (May 30, 2021 final version)
Postscript Acrobat(PDF)
Realtime JOD Version (May 8, 2021 first version)
Postscript Acrobat(PDF)


Items 1-8 were approved by straw vote on June 6, 2023.

At 98/34: the number of subbuffers to create should be specified as a “positive integer” rather than just “integer” since negative is obviously not right, and zero is not useful.
At 98/25 the Standard says that after calling MPI_CREATE_SUBBUFFERS, the new buffers are named with the name of the vector appended with the index of the buffer in the vector. The next sentence says:
“For example, the first created buffer in the V-container named bufcont has the name bufcont1.”
I believe this is a typo. The new name should be “bufcont0” since the indices start with 0.
I propose an errata entry to change the word “bufcont1” to “bufcont0” at 98/27.
It is necessary to be able to compare variables of type MPIRT_Dataspec. At 51/39 the 1.0 Standard says that MPIRT_DATASPEC_IS_EQUAL is not defined. This was an oversight/typo/mistake.
At 51/39, change “…set name, equality testing, and object decoration…” to “…set name, and object decoration…”.
Add to the table “Dataspec Operations” on page 63:
MPIRT_DATASPEC_IS_EQUAL MPIRT_Dataspec_is_equal MPIRT::Dataspec::Is_Equal
It was intended that the initial_buffer_list argument to MPIRT_BUFITER_CREATE could be given as MPIRT_CVECTOR_NULL if the user did not want to provide any initial buffers. However, this is not stated in the text of the Standard. The following errata entry clarifies this.
At 103/32, append the following new sentence to the paragraph:
The initial_buffer_list parameter may be given as MPIRT_CVECTOR_NULL if no initial buffers are desired.
In the table “Event Delivery Abstraction Operations” at the bottom of page 87, MPIRT_EVDEL_GET_NAME and MPIRT_EVDEL_SET_NAME should be added.
The following corrects and clarifies several issues in the specification for QoS for time-driven channels. Some of this could be considered part of the Error Proposal, but the issues seem too closely intertwined to separate cleanly.
The last sentence of the paragraph at 182/7 doesn’t compute. Replace the sentence with:
“The value of MPIRT_TIME_SPEC_IGNORE for the deadline indicates that a deadline is not requested.”
Delete the last sentence of the second bullet at 182/13. The correct restrictions will be described with other restrictions at the bottom of the page (see item #6 below).
Since the release time argument is required, delete the words “if one exists” at 182/16.
Change MPIRT_PERIOD_IGNORE to MPIRT_TIME_IGNORE at 182/22 and 183/5. Remove MPIRT_PERIOD_IGNORE from the tables on pages 205 and 206. This is left over from earlier drafts.
Add a new paragraph at 182/38:
If period is not MPIRT_TIME_IGNORE, it must be positive; otherwise the function will return MPIRT_ERR_INV_TIME.
6. Add a new sentence to the paragraph at 182/39:
In this case, the release_time must be non-negative and less than or equal to period; otherwise the function will return MPIRT_ERR_INV_TIMESPEC.
Add a new paragraph at 182/45:
If both release_time and deadline are of type MPIRT_ABSOLUTE, deadline must not fall before release_time. If release_time is of type MPIRT_RELATIVE, it must not be negative. If deadline is of type MPIRT_RELATIVE, it must not be negative. If any of the time specifications are incorrect or inconsistent, the function will return MPIRT_ERR_INV_TIMESPEC.
At 72/31, change “(MPIRT_TRIGGER)” to “(ref. to MPIRT_TRIGGER)”.
The following error codes are missing from the MPI/RT 1.0 Return Code Index on page 252 and the Return Codes Annex on page 244:
Page 15 23/25: Modify statement to make it not as strong as it is because even a development library may not be able to detect all errors. Replace the sentence “All of the errors defined by the standard are required to be detected and reported, in order for a library to be called development” with “A development library should make best effort to detect and report all of the errors defined by the standard.”
Change the QoS parameter in all the collective channel operation specifications from “qos” to “channel_qos”.
Page 67: Add the following new paragraph and table at line 23:
For explicit triggers and receptors, the object name must be unique over the group and GROUP_WORLD. The scope for implicit triggers and receptors is GROUP_WORLD and the scope for all receptors and triggers must match.
implicit trigger GROUP_WORLD
implicit receptor GROUP_WORLD
implicit receptor for explicit trigger GROUP_WORLD
explicit receptor for explicit trigger Same as trigger
implicit trigger for explicit receptor GROUP_WORLD
explicit trigger for explicit receptor Same as receptor
Page 67 line 33: change “no” to “do”.
It was intended that the handlers argument to MPIRT_RECEPTOR_CREATE could be given as MPIRT_CONTAINER_NULL if the user did not want to provide any initial handlers. However, this is not stated in the text of the Standard. The following errata entry clarifies this. At 74/45, append the following new sentence to the paragraph: The handlers parameter may be given as either MPIRT_CONTAINER_NULL or an empty container if no handlers are desired.
In order to clarify which events are receoverable/retryable, delete the sentence at 75/12. Insert the following sentence as the second sentence of the paragraph at 75/7:
Recoverable conditions are those implicit conditions defined as recoverable in the standard.

The enumeration type MPIRT_SYNC_FLAG is missing from the entity table at the end of Chapter 4, page 90.
MPIRT_SYNC_FLAG | MPIRT_Sync_flag | MPIRT::Sync_Flag
Page 13, line 6 change “simila” to “similar”
Page 42 line 13 change “ne” to “new”
In figure 5.8 (pg 105), all three instances of the word “buffer” should be “bufiter”.
Page 148, there is a slight formatting problem in the table – the row for MPIRT_OP is incorrect.
Errata: General Comments 11/19/00
If we use the terms V-containers, vector containers, or VB-containers it is unclear exactly what we are referencing. The standard uses the term V-container, vector containers and VB-containers as follows:
Page 34/48 -“Vcontainers are derived from MPIRT_CVECTOR_BASE and are denoted VB_containers”.
Page 38/28-30 - VB_Container classes
Page 40/35 - “a group is a VB-container of process identifiers”.
Page 35/28-30 - vector containers are denoted MPIRT_CVECTOR_BASE.
Also see buffer iterator retrieve proposal.
34/46 Change the last word of the sentence from “V-container” to “VB-container”. Delete the confusing next sentence “V-containers are derived…”
35/25 Change "whereas vector containers " to “whereas VB-containers”.
MPIRT1.1 proposal authors should also check their proposals for consistent use of V-containers, VB-containers and vector containers terminology.

Each 1.1 proposal may need a few words in the Introduction . Here are some proposed new paragraphs and additions to existing paragraphs for the Introduction.
Page 9/3 The user may specify strided buffers, allowing access to non-contiguous blocks of memory.
Page 9/19 - At any time during the non real time or real time phase, the user may retrieve the number of buffers in the buffer iterator.
Page 10/11 - such as bipartite all to all, broadcast …
Page 10/12 - A collective channel utilizing strided buffers has a natural mapping for matrix transpose and 2D transformations.
Page 10/25 A program can use a receptor to wait for an event to occur.
Page 10/38 A receptor can be created with or without handlers.
The Introduction should be cross-checked after all MPIRT 1.1 proposals are finalized to be sure that the introduction is accurate with respect to the final versions of the 1.1 proposals (the introduction is high priority since it may be the only section that a majority of interested parties read).

Add the following definition to Glossary xx/25 Strided Buffers - A contiguous buffer partitioned into fixed length sub-buffers or blocks. The stride represents the number of elements between the blocks and supports non-contiguous buffer access.
Page 3/5 During MPIRT 1.0, it was proposed and approved in a previous errata that the following words be deleted since restrictions are implied that are not necessarily accurate: " MPIRT has complete control of resources and whose", .
Page 4/33-34 Correct the sentence to read " The tear down of these objects also occurs during a non-real-time phase"
In Chapter 1, we have lists of objects and/or classes in the text. Suggest that we avoid listing specific MPIRT functionality in this chapter wherever possible. The following changes are proposed to reduce the number of places that we have to keep updating as we add new proposals over time:
7/35 Change the sentence “some examples of committable classes are…” to “Committable classes are defined for buffers, buffer iterators, channels, handlers, and event delivery. Committable objects and their classes are described in Figure 2.1.”
8/40 Add to the sentence "The standard dataspecs supported by MPI/RT are integer, floating point, double precision, logical, character, byte and time. These dataspecs and corresponding language bindings are described in Table 3.1.
The Bufiter policies are not consistent in various parts of the standard. The correct list of policies is on page 111. The following changes are recommended:
9/12-13 Delete the sentences “Currently the standard …” and “Several more policies …”. Insert the sentence “The standard specifies LIFO, FIFO, sorted and unordered user-selectable policies; in addition to several other policies if the container is a vector container.”
9/13 Delete the sentence “these mapped and ordered policies …” and replace with the sentence “The standard specifies a number of user-selectable policies as listed in the table Buffer Iterator Policies in section 5.6.”
102/30 add the words MPIRT_BUFITER_UNORDERED
Search for all occurrences of MPIRT_BUFITER_ORDERED in the standard and remove from the document.
Need to clearly associate data transfers and channels with respect to zero, one and two-sided communication. The current sentences are too disjoint and need a lead in sentence. Suggest rewording with no impact to functionality as follows:
9/35 Insert before “Figure 9.1…”, the following sentence “MPI/RT supports zero-sided, one-sided, and two-sided communication for point-to-point and collective channels.”

13/11 Remove the words “In keeping with minimizing hidden dangers”
13/16-17 Replace “because libraries can be portably written by developers” with “because portable libraries can be written by developers”"
13/35 Replace " submitted by user" with " subjected".
13/37 Remove the sentence " and thereby the target hardware".
13/38-43 and 14/1-4: With no change to functionality, remove these 10 lines, starting with the sentence “Since the admission test depends…” and replace the paragraph as follows:
A successful admission test indicates that the target platform can provide the required resources within the desired performance parameters, based on the information supplied by the user. It is the responsibility of the user to provide the correct information. The user can obtain accurate information by experimentation. Using instrumentation, the user can start with a set of plausible criteria and iterate until the desired results are achieved. The criteria used to instrument may be a variety of heuristics or a single known quantity such as minimum frame rate. Whenever the platform cannot meet the user’s requirements, the admission tests will fail. When the admission test passes, the platform/ MPIRT performance tradeoff balance has been achieved.

63/11 Add a footnote for the function MPIRT_INT64_ASSIGN64 as follows: “This function is optional”.
23/41 Change “subsequent set” to “subsequent set operation”
24/39 Insert the phrase “by the user”, after the word “FREEd”.
94/40 Change “programmer” to “user”.

Proposals for MPI/RT 1.1

Test Suite for MPI/RT

The Real-Time Message Passing Interface (MPI/RT) Test Suite is a freely available, independently developed package for evaluating the conformance of an MPI/RT implementation to the MPI/RT Standard. The Test Suite is funded by the US Navy through the Naval Sea Systems Command (NAVSEA) PMS 411.

Although the Test Suite is under development and only partially complete, the current version is available for download here. Documentation in the download package includes a Users Manual, a description of the design goals and assumptions, and an up-to-date test specification.

For additional information, contact Dennis Cottel, SPAWAR Systems Center San Diego,

Download the September 25, 2021 version (mpirt_ts_25sep2024.tar.Z)

Talks & Papers about the MPI/RT Standard



Standards efforts related to, useful with, or incorporating MPI/RT

TASP Tactical Advanced Signal Processor Effort

VSIP Forum Vector Signal & Image Processing Forum

MPI Forum MPI/RT Has its roots in MPI-1 and MPI-2,
whose standards are contained within this site.
The MPI Forum no longer meets.

Data Reorganization Standard DARPA is sponsoring an effort to look at high performance
collective data reorganization API’s and protocols, a complement and supplement to MPI/RT.

IETF PacketWay Standard
Several groups are working together to specify a high performance system area network (SAN) protocol,
including security, for interoperability, low latency, and high bandwidth. This is a key technology for
providing routing to support inter-MPI/RT implementation communication.

VI Architecture (Intel/Compaq/Microsoft)
A large group of industry interests is producing a SAN datalink layer specification (MAC + LLC)
called the Virtual Interface Architecture (VIA). VIA supports channels, and scalability.
With articulation of QoS on these channels, VIA could be an implementation mechanism
for MPI/RT channels on a SAN network, with PacketWay supporting inter-SAN routing.

Example code for the MPI/RT Standard


Contacts (Standard co-chairs): Tony Skjellum Arkady Kanevsky
Forum pages/standard document maintainer: Yogi Dandass


For more information on the base standards: