MPI/RT
Real-Time Message Passing Specification based on MPI
(MPI is the Message-Passing Interface)

This is the home page of the MPI/RT Forum. Look for regular updates,
improvements, new papers and drafts, as well as announcements of interest.

The final approved version of the MPI/RT 1.1 standard is now available in Postscript (PS) and Acrobat (PDF) formats.

The previously approved MPI/RT 1.0 standard is also available in Postscript (PS) and Acrobat (PDF) formats.

For more information on astronomy, the moon and horoscope, see here : https://www.pleine-lune.org

MPI/RT Forum meetings are open meetings, and all interested individuals, and organizations may participate by e-mail, and/or by attending meetings of the Forum. All attendees at meetings participate in informal (straw) votes, and institutions have 1 vote on official decisions if they attend two of the last three meetings, including the one then currently in session.

Here are key goals of the MPI/RT Forum.

Create integrated messaging, scheduling, and parallel programming API
together with syntax and semantics to support the emerging computational
hierarchies of node architectures and gigabit/s networks efficiently.
Expand the horizons of performance-portable real-time programming
Support multiple real-time paradigms
Enhance the performance of Messaging over MPI-1 and MPI-2
Catalyze a new generation of portable parallel applications that
require or benefit from the explicit use of time

GETTING INVOLVED AND KEEPING UP

MPI/RT 1.1 Topics

Following is a list of topics under consideration for MPI/RT 1.1:

Data-reorganization support
Collective operations (ALL_TO_ALL)
Dynamic Process Management, Parallel Client Server, etc.
Interoperation of MPI/RT Implementations
Support for Java binding of MPI/RT (other languages too?)
Full-QOS-based MPI/RT with mode changes (such as a QOS capability for mode changes)
Explicit ability to work with real-time schedulers for process scheduling
Work to support k-slack channel mode (whereas MPI/RT 1.0 is 1-slack)
Support for an IDL for MPI/RT
Fine grain, Coarse grain QOS specifications
More powerful data descriptors to allow strided and non contiguous buffers
How to handle other resources’ schedulers. Currently we have dealt with memory and network in two different ways and via completely different specs (related to 7 and 10)
Operations on containers. The old issues of propagating the operations to every object in a container (probably for the containers that contain homogeneous objects only)
Container Iterators
MPIRT_SLEEP, MPIRT_WAIT, MPIRT_WAKEUP
Channel state transition (old busy state)
MPIRT_CHANNEL_WAIT, MPIRT_CHANNEL_TEST (related to 16)
QoS Spec for channel (remove restrictions that were made for MPI/RT-1.0 at the last meeting)
Default constructors
New instrumentation metrics (Add as needed with experience)
More error returns
Remove restriction of homogeneity on bufiter parameters: dataspec and bufsize for channel endpoints
Unreliable channels
Fault tolerance, Fault handling, Fault recovery
Light-weight handlers
Half channels (I/O)
Default handlers
Group parameter scope for the commit
Mechanisms for passing in externally build schedules to MPI/RT
Logical grouping of event kinds (arithmetic on events)
PGCOs
QoS for handlers (add deadlines)
Mutable buffers (late binding of address)
Variable length buffers
Periodic triggers
Unique ID for running MPI/RT version
QoS for QoS (granularity for raising errors)
MPIRT_OBJECT_COMPARE
Operations on groups.

MPI/RT Forum Meetings

Dates	Place	Local Contact
June 5-6	MITRE, Boston	Arkady Kanevsky
July 24-25	MITRE, Boston	Arkady Kanevsky
September 25-26	MSU	Anthony Skjellum
November 6-7	MITRE, Boston	Arkady Kanevsky
December 11-12	SPAWAR, San Diego	Dennis Cottel
January 14-16	Sandia, Albuquerque	Ron Brightwell

MPI/RT 1.1 Forum Minutes

Minutes of the MPI/RT meeting Held on December 13-14, 2023 in Orlando, Florida

Attendees:

Murali Beddhu MPI Software Technology, Inc.
Dan Campbell GTRI
Dennis Cottel SPAWAR
Zhenqian Cui MPI Software Technology, Inc.
Nathan Doss LM-NESS
Arkady Kanevsky Mercury
Anthony Skjellum Mississippi State University

Five official votes (MSTI, MSU, Mercury, SPAWAR, LM-NESS)

September minutes approved after deleting item 3.

Arkady asked that proposals have dates.

Proposal describing the meaning of formal votes for MPI/RT 1.1.
FV-1: Approve the standalone proposal as is. If there are any
major changes, FV-1 must be repeated.
FV-2: Permission to integrate, with presumption of no more changes,
unless some problem is discovered.
FV-3: As part of finalization of the whole document, accept the
integrated proposal, as amended to remove errors and to add
clarifications.

a. Proposal formally approved 4/0/0

Errors Proposal v7.
a. Item 45 clarified by Dennis; output cases removed.
b. Item 1-42: FV-2, approved 4/0/0.
c. Item 43-57: FV-1, approved 4/0/0.
d. Items following 57 must be proposed at the next meeting to meet the
voting schedule approved for MPI/RT 1.1 (see item 1 above).
Error String Proposal
a. No changes since September.
b. FV-2, approved 4/0/0.
Sleep Proposal
a. Minor wording amendments were offered, not changing syntax or semantics.
b. FV-2, approved 4/0/0.
Receptor Wait Proposal
a. Minor wording amendments were offered, not changing syntax or semantics.
b. FV-2, approved 4/0/0.
Container Operations Proposal
a. Did not cast FV-1 last time because write-up was truncated.
b. Fixes offered for C/C++ bindings.
c. FV-1, approved 4/0/0.
Object is Committed Proposal
a. Small clarification made to binding using MPIRT_BOOLEAN
b. Genesis of Erratum 37.
c. Genesis of requirement to check whole document for analogous errors to
Erratum 37.
d. FV-1, approved 5/0/0
Int64 Proposal
a. Editorial corrections made; nothing major.
b. This proposal is ready to be integrated into the document now.
Container Iterators Proposal
a. Minor editorial changes, mainly to bindings were offered.
b. FV-2, approved 5/0/0.
Buffer Iterator Retrieve Proposal
a. Rename functions to avoid abbreviations
b. Discussion regarding thread safety, purported utility in practical
settings.
c. Function name changed to MPIRT_BUFITER_SNAPSHOT.
d. FV-1, approved 3/1/1.
Clique All-to-all Collective Channel Proposal
a. Modified some text.
b. Satisfied ourselves that bindings are correct; no syntactic or
semantic changes.
c. FV-1, approved 5/0/0.
Errata to MPI/RT 1.0 Standard
a. In addition to posting with MPI/RT 1.0, post with MPI/RT 1.1 document.
Certain points suspended pending 1.1 final outcomes, and will be omitted
from 1.0 posting.
b. FV-1 (including #37 as slightly amended), 5/0/0
Strided Buffer Proposal
a. Generalizing the capability cf., Figure 5.7, also pages 100-101.
b. MPIRT_BUFFER_CREATE_SUBBUFFER_STRIDED(inbuffer, byte_displacement,
element_offset, blocksize, byte-stride, nblocks, outbuffername,
outbuffer)
c. Significant discussion ensued.
d. Updated proposal will be prepared by Tony/Cui.
e. Straw vote, approved 5/0/1.
Variable Buffers Proposal
a. Proposal + fixes + actions were defined on meeting copy.
b. Updated proposal will be prepared by Tony/Cui.
c. FV-1, approved 4/0/1.
Arrangements have been made to hold the next MPI/RT Forum and Data
Reorganization Forum meetings in San Diego on the following schedule:

Wednesday, February 28th, 0800-1700 – MPI/RT
Thursday, March 1st, 0800-1700 – Data Reorg
Friday, March 2nd, 0800-1300 (incl. lunch) – Data Reorg

The meetings will be held at:

Best Western Island Palms Hotel and Marina
2051 Shelter Island Drive
619-222-0561 or 800-922-2336

There will be a meeting fee of $50 per person per day. If you plan to
pay by check, please make it payable to “SSC-SD Conference Fund”.

Please let Dennis Cottel (dennis@spawar.navy.mil) know by the morning of
February 22nd if you will attend so the hotel can be given a firm count.

Minutes for the MPI/RT Fourm meeting at MITRE, Boston on 18 September, 2023

Attendees:
Murali Beddhu MPI Software Technology, Inc.
Ken Cain MITRE
Dennis Cottel SPAWAR
Zhenqian Cui MPI Software Technology, Inc.
Yogi Dandass Mississippi State University
Nathan Doss LM-NESS
Arkady Kanevsky Mercury
James Lebak MIT Lincoln Laboratory
Jothi Padmanabhan Mississippi State University
Anna Rounbehler
Tony Skjellum MPI Software Technology, Inc.

There was a lengthy discussion regarding the future of the MPI/RT forum.
Arkady Kanevsky offered a motion to “mothball” the standard and stop
meeting. Since there was nobody willing to second this motion, an
alternate proposal was offered by Dennis Cottel, seconded by James Lebak.
This motion proposed to:
a. complete the MPI/RT 1.1 document by June 2024,
b. reevaluate the direction of the forum, and additional activities that
may be undertaken eg. Fault Handling.
c. emphasize the completion of the MPI/RT 1.1 standard as opposed to the
totality of proposals resolved.
Proposal accepted by formal vote: 6/1/1
List of proposals with a relative weight of remaining effort required
Proposal Owner Vote Effort to complete

Errors Dennis C,C 1
Error string Dennis C 0
Sleep Dennis F,C 0
Receptor wait Dennis F,C 0
Bufiter buff return Cui C 0
Int64Export Arkady F 0
Container Ops Dennis C,C,C 0
Container Iter Nathan C,C,C 0
All-to-all collective Arkady C 2 new owner: Yogi
All-to-all bipartite Arkady C 7 new owner: Yogi
Strided buffer Arkady C 7 new owner: Cui
Variable length buff Steve C 8 new owner: Cui

vote: C = Continue work to improve proposal
F = Formal vote to accept

Change hierarchy figure to reflect container iterator changes
Error String proposal:
a. Fix bindings
b. The result string is a constant string; users cannot modify or
free the result string.
c. Formal vote: 6/0/0.
Sleep proposal
a. Fix C++ binding
b. 2nd Formal vote: 6/0/0
Error proposal
a. Formal vote: 5/0/1
Receptor wait proposal
a. Fix C++ bindings
b. Sections 10.2 and 10.3, change “main program” to “program”
c. Move section 10.2 to section 10.2.2 and section 10.3 to
section 10.3.2.
d. 2nd Formal vote: 5/0/1
Container operations proposal
a. Add a proposal for container wait (all, any, some)
b. Remove search order parameter and remove error invalid mode
proposal accepted by straw vote: 5/1/1
MPIT_INT64_EXPORT32 proposal
a. Fix C++ bindings (MPIRT::INT64_EXPORT32)
b. 2nd Formal vote 6/0/0
Container Iterator proposal
a. 1st formal vote 5/0/1

Minutes for the June 6-7, 2023 MPI/RT meeting hosted by Mercury Computer Systems, Inc. in Boston

Dimitris Christodoulou SKY dimitris@skycomputers.com
Dennis Cottel SSC-SD dennis@spawar.navy.mil
Zhenqian Cui MSTI zcui@mpi-softtech.com
Yogi Dandass MSU yogi@cs.msstate.edu
Nathan Doss LMCO nathan.e.doss@lmco.com
Arkady Kanevsky Mercury arkady@mc.com
James Lebak MIT/Lincoln Lab jlebak@ll.mit.edu
Tom Murphy UMAS Lowell
Steve Paavola SKY paavola@skycomputers.com
Michael Pepe Mercury mpepe@ms.com
Anna Rounbehler arounbe@ieee.org
Jeffrey Smith Mercury jesmith@ms.com

There was a discussion on how we can make MPI/RT more popular. There was general consensus that a public domain version was needed to demonstrate the API. However,
no funding sources for this implementation exist as yet.
Dennis Cottel gave a status review of the Test Suite.
a. The test suite has been executed using the Mercury and CSPI implementations
b. Dennis is waiting for access to the SKY computer at Rome Lab
c. Dennis will put a copy of the test suite on the MPI/RT web site so others
may exercise the test suite independently.
d. SKY Computers will use the publicly released test suite on their platform
Errors Proposal
a. Arkady will post rephrased wording for paragraph on lines 23-25 on page 15.
b. Function arguments are checked in a strictly left-to-right manner based on the
LIS definition. This needs to be specifically stated for C implementations.
In C++, the object whose member function is being called is checked first.
c. Create a chapter/appendix to keep deprecated objects and constants. For example,
MPIRT_ERR_COMMITTED_OBJECT is replaced by MPIRT_ERR_OBJECT_COMMITTED.
d. The list of errors will continue to grow as the test suite uncovers additional
error conditions.
Error string proposal
a. The function should return a pointer to an error string.
b. The production mode library can return any valid null terminated string (i.e.,
does not have to return a pointer to an error string, an empty null-terminated
string is permitted).
c. This proposal should be incorporated in the document as section 1.4.4
d. Straw vote to continue work on proposal: 8/1/0
Errata straw vote: 6/0/0
Sleep proposal
a. Remove references to “thread” since a threaded model in not required by MPI/RT.
b. Replace the reference to MPIRT_TIME_INFINITE with MPIRT_TIMESPEC_IGNORE.
c. Remove MPIRT_TIME_INFINITE from the end-of-chapter summaries paragraph.
d. Straw vote to continue work on proposal: 6/0/0
Receptor wait
a. Minor wording changes.
b. Straw vote to continue work on proposal: 6/0/0
Bufiter get status proposal
a. MPIRT_BUFITER_GET_STATUS changed to MPIRT_BUFITER_RETRIEVE_BUFFERS
b. Clarify what the function returns when this function is called before commit,
after uncommit, and during the real-time (committed) mode of the MPI/RT
application.
c. The last parameter (vcontainer) of the function is an INOUT parameter.
d. The size of the vcontainer is not increased to the bufiter size. If the number
of buffers in the bufiter is greater than the capacity of the vcontainer, the
function returns error MPIRT_ERR_INV_RANGE
e. Straw vote to continue work on proposal: 6/0/0
MPIRT_INT64_EXPORT32 function proposal
a. The C++ binding has an error. The first parameter should have type const
MPIRT::Int64, not MPIRT_Int64.
b. Remove advice to users.
c. Formal vote: 5/0/0
Container operations proposal
a. The container parameter for all operations in the proposal should be IN only (not
INOUT).
b. The container must only contain objects that can be started. If the container
parameter is not a valid container the function returns MPIRT_ERR_INV_CONTAINER.
All other error returns are those raised by the channel activation function.
c. The flatten function removes duplicates without raising errors.
d. The flatten function takes a parameter of type MPIRT_TRAVERSAL_ORDER that
specifies the traversal order of nested input containers. The values of this
parameter can be MPIRT_TRAVERSE_BREADTH, MPIRT_TRAVERSE_DEPTH,
and MPIRT_TRAVERSE_INORDER.
e. The traversal order determines the relative position of the objects in the resulting
container.
f. MPIRT_CONTAINER_FLATTEN becomes MPIRT_name-of-container-class_FLATTEN.
g. The container operations, except for filter, are not applicable to Groups
h. Remove the MPIRT_ERR_DUPLICATE_MEMBER error. Make this condition the subject
of an “advise to users” paragraph.
i. Put a reference to the activate and deactivate functions in the container chapter
with the container operations.
j. Straw vote to continue work on proposal: 7/0/0
Container iterator proposal
a. Change the wording of the paragraph that states that iterators are derived from
MPIRT_OBJECT to be consistent with the other objects in the document.
b. Correct the C/C++ bindings for all functions.
c. Put the GET operations before the SET operations.
d. In the SET_CONTAINER operation, replace the last two sentences to “This operation
resets the container iterator.”
e. Clarify the meaning of “invalidate.”
f. The last paragraph before the code example, remove the text "(e.g., during a
real-time phase of processing).
g. List the meta-classes (list on page 21) over which container iterators are defined.
h. Add container interators to the object hierarchy diagram (under probe).
i. Consider adding a small introduction to container iterators in Chapter 1.
j. Deprecate old retrieve-next container operations (add to the Deprecated appendix).
k. Straw vote to continue work on proposal: 7/0/0
All-to-all collective (non-bipartite) channel proposal
a. It was decided to require that the buffers be evenly divisible by the number
of participating processes.
b. An error is returned (MPIRT_ERR_ILLEGAL_BUFITER) when the channel is
created and also when the channel is committed if the buffer size is not divisible
by the cardinality of the group.
c. Add a paragraph after the channel creation binding stating that parameter
consistency checks are performed at channel creation and commit time.
d. Straw vote to continue work on proposal: 7/0/0
Bipartitie all-to-all channel proposal
a. Add “bipartite” to glossary
b. The sentence “The order of dividing a send message into smaller messages…” needs
to be clarified.
c. A diagram explaining the data transfer is needed.
d. The two participating groups must be disjoint.
e. Add the function bindings.
f. The condition naming convention for these channels is:
object_name:group_name:rank:cond_name
g. Straw vote to continue work on proposal: 5/0/2
Strided buffer proposal
a. Reword rationale to clarify and correct.
b. Need a diagram to describe how strided buffers can be subdivided. There are
several cases. Provide examples.
c. Ensure that strided buffers work for all channel types (e.g., reduce, scatter,
etc.).
d. No existing calls will change, the proposal extends the buffer object. It does
not introduce a new object.
e. Straw vote to continue work on proposal: 5/1/2

Notes from the March 9-10 MPI/RT meeting in Atlanta.

Reviewed the updated document. DRAFT is removed from the cover.
The technical editor recommended changes (e.g., to put in common voice)
but all changes were actually made by MPI/RT editors (Yogi).
The standard, having already been voted on, is complete.

We discussed priorities for 1.1 with several folks expressing the desire
that, given the current lack of experience with 1.0, we should stick to
straightforward subjects that can be handled within a year or so.

The prioritized list:
0. errata
0.1 wait/sleep proposals which have already been voted in except
for exact wording

errors
1.5 an as yet unwritten proposal to get the two 32-bit parts from
a 64-bit time
relaxed buffer sizes and offsets
container (a) iterators and (b) operations
additional collective operations

Not making the recommended list for 1.1:
half channels
K-slack channels
generalized mode changes
data reorg capabilities

Discussion of pattern (nee helper) functions: Whether they will be part
of the standard or separate. Since we didn’t have minutes from the
December meeting (why not!) this discussion was (in my opinion) not
conclusive.

Discussion of error proposal: This included how to incorporate it
in the document: a table at the end of the chapter or do all the error
returns need to be discussed in the function text. Three returns were
renamed:
MPIRT_ERR_INITIALIZED to MPIRT_ERR_NOT_INITIALIZED (for the case when
a function is called before the library is initialized)
MPIRT_ERR_COMMITTED_OBJECT to MPIRT_ERR_OBJECT_COMMITTED for consistency,
the old name will not change but become deprecated
MPIRT_ERR_ERRONEOUS_CALL to MPIRT_ERR_UNDEF_FOR_OBJECT
Action items:

draft wording for chapter 2, etc. [Dennis]
prototype a function/error return table [Yogi]
post updated proposal and get straw vote for support [Dennis]

Discussion of variable buffers proposal: There were some minor changes and
one significant one: when receiving a buffer of variable count or offset,
both the actual count and actual offset will be set by the system. This
allows the reuse of a buffer without manually resetting its offset.

Discussion of the container iterators proposal: In addition to some wording
changes, and the note that the GET/SET_CONTAINER functions are missing,
it was decided that duplicating an iterator also duplicates the current
position so that the two iterators are indistinguishable after the DUP.
This is so that a copy of an iterator can be made at the current point in
the iteration and returned to later.
Action items:

send your notes to Nathan [all]
rewrite and resubmit [Nathan]

This led to a discussion of whether a DUP of an object gets you a copy
of the parameters at that moment, or the parameters as they were when
the DUPed object was originally created. Several felt the document said
the latter, while the consensus was that the former was intended – or
at least wanted now. Changes will be made to clarify this and included
in the errata.
Action item:

write up the errata [Tony]

Another errata item noted: all figures (and tables?) should have numbers,
and there should be a Table of Figures.
Action item:

write up the errata [Tony]

Discussion of container operations: The difference between regular and
atomic version of all were revisited. The result was that the proposal
would be revised as follows:
there will be a single version of the call for each (remove the atomic
definitions)
the order of operation is determined by whether the container is a
vector or set: if vector, user specifies; if set, implementation specs.
activate, raise, and start: order means getting the buffer from the
bufiter and enqueing; QOS determines subsequent execution
deactivate: order means wait for each to finish in order (note that
deactivate is essentially a blocking call)
Action item:

rewrite and resubmit [Dennis]

Discussion of additional collective channels: Only the important cases should
be addresses to avoid delving into Data Reorg issues which will be a separate
proposal one of these days. The most important cases are
all to all clique
all to all bipartite
exclusive scatter (sender doesn’t receive any)
exclusive gather (receiver doesn’t send any)
The last two are special cases of the second. In the interest of minimal
additions and short time, it was decided to do the first two cases with some
restrictions: the two groups must be disjoint, and (like current gather and
scatter) the parts have to be evenly divisible by N processors.
Action item:

write up a proposal [Arkady]

New proposal wanted to retrieve the current buffer count and list from a
buffer iterator. [Action: Cui]

MPI/RT Meeting, September 24, 2023, held at Mercury Computers

The following attendees got down to business at about 9am:

Dennis Cottel, SSC-SD, dennis@spawar.navy.mil
Dimitris Christodoulou, SKY, dimitris@sky.com
Nathan Doss, LM/GES, nathan.e.doss@lmco.com
Mike Melandez, Mercury, melendez@mc.com
Arkady Kanevsky, Mercury, arkady@mc.com
Anthony Skjellum, MPI Softtech, tony@mpi-softtech.com

The purpose of the meeting was to review changes made to the document at
the meeting two weeks ago, Sep 8-10, held at SKY. The document reviewed
is dated September 24, 2023 as it was printed in the wee hours on the
day of the meeting.

Several items in the Glossary were revised, as was Figure 1.1, to make
them consistent. The revised Glossary was accepted by a formal vote of
5/0/0.

A description of the “name_of_class” notation was added to 1.1.5 and 2.1.

We reviewed essentially all of Chapter 1 since there had been numerous
changes, and accepted it by a formal vote of 4/0/0.

In Chapter 2, for consistency, MPIRT_KEYVAL_DELETE_FN was renamed to
MPIRT_KEYVAL_FREE_FN with all the attendant changes.

Also in Chapter 2, there was a lengthy and arcane discussion of the C++
API with the result that “constructors” are public, not private as
currently shown in the example in 2.7. The minutes from the last
meeting asked that a new section on C++ be added and this had not been
done (as this continuing discussion pointed out). The new section will
be added.

We reviewed primarily the marked changes on Chapters 3 through 15 with
only minor edits and cleanup, including picking up some previously
mandated changes that hadn’t made it into the document.

We adjourned at 2pm.

Minutes from the June 8-10 2023 MPI/RT Forum meeting that was held at LM-GES in Moorestown, NJ.

Attendees:

Arkady Kanevsky (co-chair) Mercury Computers arkady@mc.com
Dennis Cottel SPAWAR (SYSCEN, San Diego) dennis@spawar.navy.mil
Randall Judd SPAWAR (SYSCEN, San Diego) judd@spawar.navy.mil
Nathan Doss LM/GES nathan.e.doss@lmco.com
Tom McClean Lockheed Martin (LM/GES) Thomas.P.Mcclean@lmco.com
Shane Hebert MSTI shane@mpi-softtech.com
Michael Grieco ASEC/NSA mjgrieco@jswg.org
Steve Paavola SKY Computers paavola@sky.com
Anthony Skjellum (co-chair) MSTI tony@mpi-softtech.com
James Lebak (minutes taker) MIT/LL jlebak@ll.mit.edu
Yogi Dandass MSU yogi@erc.msstate.edu

SUMMARY
There was 1 official vote on a clarification to MPI/RT-1.0 document.

A. MPIRT_INIT is a required synchronization point for all processes. Formal vote passed 7/0/2.

There were several straw votes on the MPI/RT-1.0 document.

The name for the states should include the name of the object. This means that instead of
“ACTIVATED” it should be “MPIRT_CHANNEL_ACTIVATED”. See issue 9. Vote passed 7/0/2.
The changes for issue 21. NULL objects for all types. New operation to find out if an object is NULL
of any type. Dup of NULL object refers to the same object. Vote 7/2/2.
The tables of all possible return codes for each function should not be done for MPI/RT-1.0 document,
but should be done in MPI/RT-1.1 document. Vote 8/0/0.
No clarification is needed for best effort quality of service for triggers. Vote 9/0/2.
Channel activation is not allowed if in-bufiters for all channel endpoints are shared with other
channels. Vote 6/0/5.
The changes for issue 27. Clarification for Commit for Best Effort QoS specification. Vote 8/0/3.

There were several formal votes on MPI/RT-1.1 proposals.

The Sleep proposal with changes passes the official vote 7/0/1.
The Receptor Wait proposal with changes passed the official vote 8/0/0.

I. The meeting came to order at 8:30am on Tuesday June 7.

II. The Minutes of the April meeting where approved 8/0/1.
The names of three proposals for relaxing data descriptor requirements will be added to the
minutes. Change Morristown to Moorestown NJ.

III. During most of the meeting the changes to MPI/RT-1.0 document were discussed.
The following list of 46 items for change requests accumulated since the March meeting was presented.
Majority of the items is done already. The remaining items and the changes agreed at this meeting will be
done prior to the next one.

Go through entire document and use \state & \cond macro for the states and conditions of all objects.
Done
In introduction chapter (chapter 1) write sections 1.3.4 (Erroneous programs) and sections 1.3.5
(Implementation libraries).
Done
Remove “draft” from the cover page (last one to do!)
Remains open, the last one to be done
C++ binding changes
Done, see issue 19 for more changes for C++ binding approved by the meeting
Check C bindings (especially for Containers)
Open. Add in object chapter that for MPIRT_CONTAINER users need to do typecasting to the specific
container time (CSET, CVECTOR, GROUP) for C binding. No typecasting needed for C++.
Check how we use “transmittable” in glossary, in intro, in dataspec chapters.
Open
Add to the preface “how to read the document” (what advises mean, naming conventions, and so on)
Done
Resize the figures
Done. Figure 2.1 on page 19 is too large. Reword citations after figures 8.1, 8.2.
Decide if we should add the name of the object before the state (another words instead of
ACTIVATED it will be MPIRT_CHANNEL_ACTIVATED)? If decision is positive then do global
change for all state names.
The straw vote passed 7/0/2 for longer names. The current document (June 7 '99 version on web)
already reflects this change.
Add examples to the final document.
Everybody agreed that we need examples. There was a discussion on whether we should add them to
the MPI/RT-1.0 document, MPI/RT-1.1 document, put them on the mpirt web page or in a separate
document. It was pointed out that examples are not part of the document and are not subject to the
vote. At the end it was agreed that the examples will not be a part of MPI/RT-1.0 document but will be
a part of MPI/RT-1.1 document and will also be placed on the web. All the vendors should check if
they can “donate” examples. All examples placed on the web and in the document should be tested on
the vendor’s implemented MPI/RT. MPI/RT editors will try to ensure that they also run on all existing
MPI/RT implementations (Mercury, Sky, & CSPI).
Define MPIRT_ERR_OBJECT_NOT_COMMITTED error which will be return by a commitable
object for operations that require the object to be committed. Add this error for all appropriate
functions.
Error is defined in the latest draft. Error description needs some rewording.
For each function define a table with all possible status return values and under what conditions each
value is return. (May be just add these tables at the end of each chapter rather then in each function
description) Synchronize with Annex A and the first (status returns) index.
It was agreed that this should not be done in 1.0 but should be done in MPI/RT-1.1 document. The
straw vote was taken and passed 8/0/0.
Define a generic error which will be return in all cases where a specific error name is not defined.
Document all these places and generate a list of suitable error names for all these cases.
It was decided not to add a generic error. The existing error MPIRT_ERR_INV_OBJECT will be
changed to MPIRT_ERR_INV_ARGUMENT to handle generic invalid parameter to MPIRT operation.
Clarify what data is in the out-bufiter if the transfer is not successful. Clarify how an implementation is
allowed to abort the transfer.
Data in the “last” buffer is undefined if the error is returned. The buffer is still moved from input to
output buffer iterator in the case of error. If QoS is pure priority or best effort the
MPIRT_ERR_TRANSMISSION is returned. The related condition is defined for the channel object so
that users can define a receptor and handlers to deal with it. For all other QoS specifications
(deadline, timeout or stop event) the existing QoS error should be returned. For MPI/RT-1.1
investigate what can be done with transmission error status. Remove “Abort” (for channel
transmission) from the document.
Can objects (commitable) be decorated at any time? After Commit?
Yes, objects can be decorated at any time. Add a note to the text on that.
Elaborate on scatter_channel (as it is done for gather_channel).
Done. Add a sentence that the receive buffers for all channel endpoints are the same. Add analogous
sentence for gather channel also. Add the figures for the definitions for collective channels from the
MPIDC MPIRT tutorial.
Define a missing CVECTOR_NULL in text.
Done
All operations should return int. Fix returns for operations in 3.2.3.
Done
Dup should be defined for all object not just leaves of the object hierarchy (implications for bindings)
The discussion of this issue quickly centered what to do with operations that take non-leaf parameters.
It was agreed that the current DUP definitions are wrong. The current DUP allows both copy
constructor and clone. For leaf classes DUP operation will be define explicitly and no type casting is
needed. Add advice to the users that for some operations on virtual classes type casting may be needed
for some language bindings. For functions that takes non leaf type parameters (CSET_INSERT,
CVECTOR_REPLACE, DUP, FREE) type casting is needed. Copy constructor should be described
separately from DUP. Finally the following solution was agreed upon:
a. Default constructor which initializing to object_NULL (of appropriate type). [int =
obj1.create(.) ? user. int MPIRT::object::create(.);]
b. Create fills it (C++ returns error code just like C)
c. Dup is not copy constructor. (C++ returns error code just like C) [int=obj1.dup(obj2) ? user
use; int MPIRT::obj1::DUP(obj2) ? C++ binding] Dup is defined for MPIRT_OBJECT. All
other objects inherit it. Type casting is needed even for leaf. Users can either call dup on leaf
object type directly or use dup of virtual class with type casting. No ==.
d. Explicit free call separate from destructor. Use the same explanation as for constructors.
[Rationale: No exception calling, error returns code instead. Exception handling too expensive.]
For C++ add Free before destructor. Add a separate section for C++ for constructors and
descructors.

At the second day it was pointed out that C++ provides copy-constructor and op = automatically. That
means that an MPI/RT object is handle to the actual object and that = and copy-constructor do byte-
by-byte copy of an object handle and the new object is IS_EQUAL to the original object as defined by
MPIRT_OBJECT_IS_EQUAL operation.

Define MPIRT_OBJECT_NULL. (static member of base class)
Done
What are the NULL objects for non-leaf classes? Do we have them? Do we need them?
Yes, we need NULL for all objects. Add new functions MPIRT_OBJECT_IS_NULL(obj,flag) which
returns that the flag is MPIRT_TRUE if the object is a NULL object of any type.
MPIRT_<object_class>_NULL constant has object type. NULL objects have object name of empty
string “”. Duplication of NULL object of any type refers to the same NULL object. This means that
MPIRT_OBJECT_IS_EQUAL for a copy of the NULL object and NULL object will return
MPIRT_IS_EQUAL. The straw vote was taken for these proposed changes, which passed 7/2/2.
What should is_equal operation return? (binding implications). What does it return for “comparison”
for object_null, channel_null, pt_channel_null?
See previous issue.
What is the meaning of the group WORLD if INIT is not a synchronization point?
MPIRT_INIT is now a required synchronization point for all processes. Formal vote passed 7/0/2.
Should we add an error MPIRT_ERR_TIMEOUT for commit to return when one of the group
WORLD members was not created?
It was a general consensus that timeout for commit is useful and desirable. However it was decided to
postpone this until MPI/RT-1.1 where is part of the QoS of commit issue and is a part of the RT mode
change proposal.
Clarify which channel will transfer a buffer for shared bufiter for BEST_EFFORT QOS.
This discussion was spread over two days. There are two real issues: one is what guarantees does
implementation provides for best effort (and priorities), and which channel should transfer a buffers in
the case of shared bufiters.
For the first question the agreement was reached that if the resources are available then the progress
shall be made according to the quality of service. This means that for the case of priorities, if
resources are available the highest priority channel will transfer among all channels that are ready to
transfer a message. In the case of equal priority or best effort (the lowest priority) any channel can
make progress if resources are available and no higher priority channels are ready and if no
scheduled messages exist for time-driven QoS spec. Replace “infinity” with bounded deadline (can be
very large) in the definition of best effort on page 165. Clarify that if resources are not available then
no progress will be made for a channel message transfer for best effort and priority QoS
specifications. The lower priority channel can be delayed indefinitely if resources are not available.
Remove advice to the users on page 165.
For the second question there were several sub-issues to clarify. First, the clarification will be made to
description to the START operation on page 149 to unsure that the buffer is removed from the input
bufiter when the start returns. The completion of start is instantaneous and does not mean that the
transfer had started. The situation with ACTIVATED is more complicated. The small state transition
diagram for activated case was accepted. There are three states inside current activated state: Ready,
Xferable, and Xfer data. When the buffer is inserted in an activated channel in-bufiter the channel
transitions into Cferable. If some other channel (or user) removes the buffer from the bufiter and the
input bufiter becomes empty the channel transitions back into Ready state. From Xferable when all
other channel endpoints are in Xferable the channel endpoints all transition into Xfer data and
transfer message(s) over the channel. At the completion of the transfer the channel transitions into
Ready state again. This clarification will be imbedded into the figure 9.1 on page 153.
The table was created which will be added to the document that defines what happens for channel
endpoints in the case of started/activated and in the case of shared and not shared bufiters. All the
cases are already defined except the case of shared bufiters and activated channel endpoints. For the
cases of activated channel endpoints if only one side: send or receive has shared in-bufiter this is
equivalent to get and put models respectively. The case when channel endpoints are activated and their
in-bufiters are shared is NOT allowed for MPI/RT-1.0. This ensures that there are no deadlocks. The
last statement had a straw vote that passed 6/0/5.
Clarify the meaning of MPIRT_QOS_BEST_EFFORT for triggers.
No special clarification needed. Straw vote 9/0/2.
What is the requirement for MPIRT_QOS_BEST_EFFORT from COMMIT?
Commit succeeds even though there is no “bandwidth” for that best-effort channel?
There is a time at the end of a very long period?
No need for changes. Issue 25 already covers it. Straw vote 8/0/3 passed.
Clarify the requirements for a user for putting/getting buffers to/from bufiters for barrier channel. Can
the bufiter_null be used for that type of channel?
Open
Check Anna’s email comments that she sent to us.
a. Anna’s glossary comments
b. Anna’s comments on commit and data transfer
c. Anna’ detailed list of comments
Done
Fix MPIRT_NULL problem in MPIRT_PROBE_CREATE
Done
Agree on the format of for C++ for QOS and DATASPECS (should be a part of 4).
Done. There are various consistency problems with indexes. Define in footnote what bold page means
(definition).
Add underlying assumptions of reliability and ordering of messages (appropriate to QoSs)
Done
In 2.2.5 remove that group is “read-only”. Remove all “read-only” in other places.
Done. Add that GROUP_WORLD, GROUP_SELF and GROUP_EMPTY are all constants and are
read-only. All constants are read-only.
Add definition of reliability to Glossary.
Open
Clarification for MPIRT_CONTINUE return for the last synch handler
Done
Fix …COND_QOS_FAILURE and …COND_FAILURE as well
as analogous cond for triggers and receptors. Check against analogous
cond for channels.
Done
Clarify what is rembuffer for CVECTOR_CREATE. A “semiformal”
votes by email agreed on MPIRT_BUFFER_NULL.
Done
Clarify that recoverable and retryable operation are only for
standard defined operations and not user-defined ones.
Done
Fix MPIRT_Int64_assign32 from int to unsigned int.
Done
Clarification and advice to users how to create and manipulate groups.
Done. More editorial changes needed. Remove the second half of the first paragraph of 2.5.2 (group
manipulation are defined - inherited from vector container). Change rationale to regular text. Define
“hole” as NULL object. Add a note that instantaneously a group can have duplicates but none at
commit time.
Clarify the effect of changes (including free) of the original buffer on buffers that are derive from it
using buffer partitioning operations.
The original “big” buffer must be committed with all its children. All the buffers over the same
SYSTEM_MEM_ALLOC must be committed together (user requirement). Free of buffers over the same
memory must be done in the opposite order. Clarify that system allocated memory is allocated at
commit time and free at uncommit time (and commit if a buffer not present). The characteristics of the
original buffer (first to specify shared memory) cannot be changed if there are buffers derived from it
(either dup without base change or buffer partitioning). The memory linkage is always to the first
buffer that is created over the same memory using SYSTEM_MEM_ALLOC. Dereferencing (changing
base address) of any of the intermediate buffers over the shared memory has no effect on its children
who are still referencing the original buffer memory. Add an error if the system alignment for the
dataspec is broken by the change of dataspec or base address.
Add definition of reduce operation (from MPI-1).
Open
Clarify receptor duplication operation and event name in particular.
Done. Copy the new text from section 4.2.2 (common EVDEL operations) to section 4.2.4 (receptors).
Other small editorial changes.
Fix event name problems
There is a problem with uniqueness of event names. The event delivery abstraction scope for MPI/RT-
1.0 is GROUP_WORLD. That means that the event name must for be unique over GROUP_WORLD.
The forum considered a proposal of adding a group to the event name for implicitly generated events.
For explicit events it is a user responsibility to ensure the uniqueness of the name. The advice to the
user should be added to clarify this. As part of this discussion the decision was made to add a
requirement that the name of the group object must be the same for all members (and users- see
discussion below) of the group (processes of the group). This will simplify an implementation job of
matching all the processes of the group to ensure that the group definition is correct. The old
requirements of the same task address in the same order in the group (vector container) remains. A
new error MPIRT_ERR_GROUP_INCONSISTENT will be added. This error covers both duplicate
group names and that not all group processes created the right group.

The old specification for the channel object implicitly generated events is a STRING_NAME of type
“channel name”:rank1:rank2:“condition name”. This requires that the channel name is unique over
the GROUP_WOLRD. The same channel name must also be unique over the group over which the
channel is created. There was a discussion to change the format to group:rank:object type:object
name: condition for all implicitly generated events. After long discussion it was agreed that it is not
needed for MPI/RT-1.0 fix.
All objects that can generate implicit events except channels are generate within one process and
should have a unique name within this process. The rank is part of the event name for them so we can
think of them as scoped over the group GROUP_WOLRD. Channels are scoped over the group over
which they are defined. It is user responsibility to ensure that if there are receptors for the channel
events then the channel name must also be unique over group world and hence unique in the process
where endpoint is defined. Thus, channel event names become analogous to the other object event
name if “rank 2” is dropped. The receptor’s process for channel events does not have be a member of
the group over which the channel is defined.
The following two resolutions were made: “rank 2” is not needed for channel event names. When
defining an implicit receptor, the scope passed to the receptor needs to be the same as the group over
which the object is defined, and the receptor process does not have to be a member of that group. This
means that if a user would like to have an implicit or an explicit receptor for an implicit event, the user
needs to create a group over which the object generating the implicit event is defined (group of a
channel and MPIRT_GROUP_WORLD for the rest) and the user needs to pass that group as an input
parameter to an explicit receptor or a QoS specification for implicit receptor.
Define object names for object created by implementation (e.g. GROUP_WOLRD, GROUP_SELF)
Done
Clarify that buffers in the initial buffer list are inserted into the buffer iterator at commit time.
Done.

V. At the next meeting the full review of the MPI/RT-1.0 document will take place with the formal
votes.

VI. The next Forum meeting will take place at Sky Computers on Sept. 8-10 W-F (half day Friday)
(no Data Reorg). Data Reorg will take place at HPEC.
The meeting after that will take place at Mercury Computers on Oct 19-22 Tu-Fr. MPIRT meeting
will be 2 days Tu-Wed. Data Reorg 1and a half Th & Fr. Friday is half day.
The final meeting of the year will be in San Diego Dec. 14-17 Tu-Fr. MPI/RT- 2days, Data Reorg
1 and a half. Friday is half day.

VII. The k-slack proposal was discussed.
There were several requests to clarify things. First, what is the relationship between QoS and k-
slack? Second, the current proposal draft is geared toward 0-sided communication. Please,
defined what happens for 1- & 2-sided models. Are multiple starts allowed with k-slack? What is
the definition of WAIT for the k-slack channel? The same question for TEST, and channel
conditions and implicit events. Add functionality so that a user can “wait” for a buffer associated
with a particular message transfer (transaction).
As part of this discussion the Forum asked for a clarification for MPIRT_CHANNEL_WAIT in
MPI/RT-1.0 document on page 151 lines 13-27.

VIII. Container Iterators proposal was discussed.
The container argument should be flat (no nested containers of any type) and contains only
objects of the type on which the operations can be performed. Two new functions FLATTEN aand
FILTER should be added to help with that. The order in which the operation should be performed
on the objects in the container is undefined and implementation is free to apply the operation on
objects in parallel. New errors will be introduced if the input container is not flat or contains
objects of the wrong type.

IX. Sleep proposal was discussed.
What happens on mode changes? Sleep should return when Commit or Uncommit is called. The
return code should specify why returned. If specified time is an absolute time which had passed an
error should be returned. The formal vote on the proposal had passed 7/0/1.

X. Receptor wait proposal was discussed.
What happens on mode changes? Receptor_Wait should return when Commit or Uncommit is
called, or when receptor goes away. The return code should specify why returned. The formal vote
on the proposal had passed 8/0/0.

XI. The testing suite for MPI/RT-1.0 was discussed.
The Forum should endorse it. For outreach a new webside “next” to mpirt web site should be
created with cross pointers. What should be done for the first spec? By december it has only time
for a small subset. What functionality should it include? Initially in C without eliminating C++.
Initially should not concentrate on what happens for erroneous programs. Meta standardization:
run command, error formats, hidden flags and access to implementation internals. Debug vs.
production mode libraries.

XII. The proposal for relaxing data descriptor requirements was discussed.
Change “get” to “retrieve”. Change MPIRT_INT to “int”. It should represent the number of
elements of the buffer dataspec. Change “length” to “count”. Offset should be non-negative and
is from the start of the buffer. The current proposal is geared to ptchannel, specify what effect
these changes have for collective channels. Define relationship between buffer count + offset to
the bufiter count. All dataspecs should match. Specify what size buffers can be inserted in bufiter (
for variable length and offset) and which part of the buffer is transferred (send side, receive side).
The count of the received buffer is modified by the channel to reflect the received message size.

XIII. Two new topics where added to MPI/RT-1.1 list of topics:
1. Allow Init, Finalize and new Init.
2. Helper function for freeing all buffers over specified system_mem_alloc.

XIV. There was a small group discussion while Data Reorg was in preparation. There was a discussion
on old Compare operation and object equivalency. Users themselves can decide what equivalency
mean for each object type. Two objects for which it is hard to do are buffers and task_addresses.
The following decisions were made to handle that:
1. For MPI/RT-1.1 we need MPIRT_BUFFER_GET_MASTER (buffer, &master, &offset)
operation which returns the first buffer that requested the shared system_mem_alloc memory.
The offset will be in bytes to handle the situation when dataspecs where changed in the tree of
created buffers over the same memory. Hence, specifying the offset in dataspecs is not
possible.
2. For MPI/RT-1.1 add MPIRT_DATASPEC_SIZEOF(dataspec) operations to find the size for
dataspec in bytes.
3. For MPI/RT-1.0 remove DUP and FREE operations for TASK_ADDRESS.

Minutes from the April 7-8 2023 MPI/RT Forum meeting that was held in Bedford, MA.

These minutes were approved with a vote of 7/0/1 by the forum during the June 2023
meeting.

Attendees:

Arkady Kanevsky (co-chair) Mercury Computers arkady@mc.com
Charles R. Petersen AF Research Lab, Rome pedersen@rl.af.mil
Tony Lee NAVAIRWARCTR Pax River leeat@navair.navy.mil
Ken Cain (host) MITRE kcain@mitre.org
Manoj Apte MSU manoj@erc.msstate.edu
Dennis Cottel SPAWAR dennis@spawar.navy.mil
Nathan Doss LM/GES nathan.e.doss@lmco.com
Tom McClean Lockheed Martin (LM/GES) Thomas.P.Mcclean@lmco.com
Anna Rounbehler Self arounbe@aol.com
Henk Spaanenburg Mercury Computers hspaanenburg@mc.com
Shane Hebert MSU shane@erc.msstate.edu
Zhenqian Cui MPI Software Technology Inc. zcui@mpi-softtech.com
Raghu Angadi MSTI angadi@mpi-softtech.com
Michael Grieco ASEC/NSA mjgrieco@jswg.org
James Lebak MIT/LL jlebak@ll.mit.edu
Steve Paavola SKY Computers paavola@sky.com
Dimitris Christodoulou SKY Computers dimitris@sky.com
Rick Massary NAWCAD massaryrj@navair.navy.mil
Chris Lirakis BBN Technologies clirakis@BBN.com
Matt Daily BBN Technologies mdaily@bbn.com
Thomas McMahon MSTI thom@mpi-softtech.com
Srigurunath Chakravarthi MSTI ecap@mpi-softtech.com
Anthony Skjellum (co-chair) MSTI tony@cs.msstate.edu

SUMMARY
There wers 1 official vote on a clarification to MPI/RT-1.0 document
and several straw votes on the MPI/RT-1.1 proposals.

The meeting came to order at 8:30am on Wednesday February 3.
The clarification was requested on inserting and removal of NULL element
into CSET.

The following clarification was officially voted on:
�Insertion of NULL object of any type into CSET does not do anything (no op)
and returns MPIRT_SUCCESS without any changes to CSET.
Removal of NULL object of any type from CSET returns an error
MPIRT_ERR_CSET_MEMBER_NOT_FOUND.�

The official vote was 11/0/0.
The proposal on Container Iterators (version 5) (Nathan) was discussed.
The three options (two stated in the proposal and one by Greg email)
of what happen to the container iterator while the container is modified where considered.
The straw vote was taken to remove options 1 and 3 and leave only version 2.
The vote passed 13/0/4.
It was agreed to leave MPIRT_CSET_RETRIEVE_NEXT for backward compitability.
The whole proposal was straw voted on and passed 14/0/2.
Nathan agreed to convert the proposal into a chapter which will be merged with MPI/RT-1.0
document (after all the clarifications will be done) which will be a base for MPI/RT-1.1 document.
The proposal on Receptor Wait (version 3) was presented by Dennis.
The following changes were agreed to for the proposal:
A new section 14.5 for �timed waiting� will not be added.
The �receptor� parameter is no longer optional, so MPIRT_RECEPTOR_NULL can not passed into
MPIRT_RECEPTOR_WAIT. Separately, There was agreement to create an MPIRT_SLEEP functionality.

The modified proposal passed a straw vote 18/0/3. Dennis agreed to update a proposal and identify all
the changes need for it in MPI/RT-1.0 document. The bulk of the proposal will go into Chapter 4.

The sleep proposal was discussed.  Functionality identical to the

MPIRT_RECEPTOR_WAIT for the NULL receptor was agreed upon. The MPIRT_SLEEP function
will provide that.

The straw vote on the proposal passed 18/0/1. Dennis agreed to update a proposal and identify all the
changes need for it in MPI/RT-1.0 document. The bulk of the proposal will go into Chapter 14.

The Unreliable channel proposal (version 1) by Arkady/Leonard was briefly discussed.
It was agreed that a clear definition of what an unreliable channel is is needed
as well as the requirements for the delivery of a message on it. The meaning of QoS for this type of
channel also need to be clarified. No vote was taken on this proposal.
The Half channel proposal by Shane was discussed.
It was agreed that the wait_function is not needed. The stop_function was renamed abort_function.
The need of xfer_complete for the status of the transfer completion was discussed and will be added
for the next version of the proposal. The discussion of relationship between a user, MPI/RT and device
vendor of the �remote� endpoint of half channel was discussed. It was reduced to “who provides the
half channel functions that are part of the half channel definition.”
Shane agreed to update the proposal for the next meeting. No votes were taken for this proposal.
The k-slack channel proposal by Shane was dicussed.
The relationship between this specification and QoS was considered. A clear indication how this
functionality will benefit the MPIRT users was requested. The need for this functionality for the
MPI/RT-1.1 vs. MPI/RT-2.0 was also considered. Shane agreed to investigate these issue further for
the next meeting.
The RT mode changes mode proposal (version 1) by Arkady was discussed.
It was agreed that we will keep existing COMMIT and UNCOMMIT operations but they will be
written in terms of the RT mode functionality for a single transaction at a time form.
Since some object should be passed around from one mode to another without really being used but
simply for their state preservation a FLAG for identifying these object in the �mode container� should
be added. No sets operations should be allowed on all objects after admission test (even if they are not
used in the current RT mode), since these objects promised to be unchanged for future transits.
Since admission testing is a separate functionality now an Admit for RT mode should be added.

There was a long discussion on a situation when an object is in a certain state on starting RT mode of
the transition but need to be at a different state at the end of the transition for the new RT mode. The
prime example of it is a buffer iterator initial buffers which may be in some other buffer iterator when
transition starts. It was agreed that for this proposal we should state that this situation is possible and
that users can prevent it by having empty initial buffer lists for buffer iterators. Users can insert buffers
into buffer iterator themselves after transition.

It was pointed out that the MPI/RT-1.0 should clearly state that the buffers from the initial buffer list
are required to be in the buffer iterator only after COMMIT.
All parts of the �global� object must specify the same information.
Admission_test is synchronous and collective.
Arkady agreed to update the proposal for the next meeting. No votes were taken on this proposal.

Three proposals for relaxing data descriptor requirements between buffer and buffer iterator and
across a channel were presented by Steve and Shane. The proposals are as follows:
a) relax the restriction that buffer dataspec and length must match the bufiter specs,
b) allow variable length data transfers, and
c) allow the transfer start address to be offset from the start of the buffer.
Each proposal was first discussed in isolation. The implication of one proposal on the others and
possible ,configurations of proposals were discussed. It was agreed that more details are need for that.
Steve agreed to provide these for the next meeting. No votes were taken on this proposal.
The future meeting were considered. The next meeting was scheduled for June 8-10, Tuesday �
Thursday, with the last day half day meeting in Moorestown New Jersey. Nathan Doss will host the
meeting and will provide all the information. (Data Reorg will take place on Thursday second half of
the day and Friday � June 10-11.)
The other future meeting will be held in late August, October, and December.
There were many clarifications and corrections requested for the current MPI/RT-1.0 document:
A. Defined an error that will be returned for commitable objects operations that are only defined for
when an object is committed in the case when the object is not committed, MPIRT_OBJECT_NOT_COMMITTED.

B. Document which errors each operation can return.

C. Clarify what data is in buffer in the out-bufiter if the transfer not successful.
How is implementation allowed to abort a transfer.

D. Can objects (commitable) be decorated at any time? After Commit?

E. Elaborate on scatter_channel (as it is done for gather_channel).

F. Define a missing CVECTOR_NULL in text.

G. All operations should return int. Fix returns for operations in 3.2.3.

H. Dup should be defined for all object not just leaves of the object hierarchy.
(implications for bindings)

I. Fix C++ bindings. Does it have extra * or not? Check C bindings especially for container
operations

J. Define MPIRT_OBJECT_NULL. (static member of base class)

K. What should is_equal operation return? (binding implications). What does it return for
�comparison� for object_null, channel_null, pt_channel_null?

L. What is the meaning of the group WORLD if INIT is not a synchronization point?
Should we add an error MPIRT_ERR_TIMEOUT for commit to return when one of the group
WORLD members was not created?

M. Clarify which channel will transfer a buffer for shared bufiter for BEST_EFFORT QOS.

N. Clarify the meaning of MPIRT_QOS_BEST_EFFORT for triggers.

O. What is the requirement for MPIRT_QOS_BEST_EFFORT from COMMIT?
Commit succeed even though there is no �bandwidth� for that best-effort channel?
There is a time at the end of a very long period?

P. There are several others that were send electronically to the exploder and not presented at the
meeting.

The editorial board chaired by Tony and Arkady will clean up MPI/RT-1.0 document and will
Present an update document at the next meeting. The binding issue will be resolved within two weeks.

Mike Greco provided a written update on RT-CORBA and related OMG activities.

Minutes from the Feb. 3-5 2023 MPI/RT Forum meeting that was held in San Diego, CA.

Attendees:

Arkady Kanevsky (chair) MITRE arkady@mitre.org
Manoj Apte MSU manoj@erc.msstate.edu
Randall Judd SSC-SD judd@spawar.navy.mil
Dennis Cottel (host) SSC-SD dennis@spawar.navy.mil
Leonard Monk CSPI lmonk@cspi.com
Nathan Doss LM-GES nathan.e.doss@lmco.com
Shane Hebert MSU shane@erc.msstate.edu
Darwin Ammala MSTI dammala@mpi-softtech.com
Clay Taylor, Jr MSTI cdtaylor@mpi-softtech.com
Michael Grieco ASEC/NSA mjgrieco@jswg.org
James Lebak MIT/LL jlebak@ll.mit.edu
Steve Paavola SKY paavola@sky.com
Rick Massary NAWCAD massaryrj@navair.navy.mil
Rob Ginn NAWCAD rob@sun701.nawcad.navy.mil

SUMMARY
There were no official or straw votes.
The following action items were discussed and
assigned for the proposal writing:

A. Arkady - RT-Mode changes (topic 6)
B. Shane - K-Slack channels (topic 8)
C. Steve - Data descriptors (topic 11)
D. Dennis - Sleep, wake up, wait on (topic 15)
E. Shane - Bufiter Dataspec homogeneity (topic 22)
F. Arkady & Leonard - Unreliable channels (topic 23)
G. Shane - Half channels (topic 24)
H. Steve - Relocatable buffers (topic 43)
I. Steve - Variable length buffers (topic 44)
J. Leonard & Arkady - Unique ID (topic 46)

H. Arkady - Update channel state transition diagram and write a description for
it for MPI/RT-1.0.

The meeting came to order at 8:30am on Wednesday February 3.
The updated list of action items for the MPI/RT was discussed.
The following items were added to the list over the three days:
a. Light-weight handlers
b. Half channels
c. Default handlers
d. Group parameter scope for the commit
e. Mechanisms for passing in externally build schedules
to MPI/RT
f. Logical grouping of event kinds (arithmetic on events)
g. PGCOs
h. QoS for handlers (add deadlines)
i. Mutable buffers (late binding of address)
j. Variable length buffers
k. Periodic triggers
l. Unique Id for running MPI/RT version
m. QoS for QoS (granularity for raising errors)
The topic of real-time mode changes was discussed (topic 6).
First, the short summary of the history of the topic
was presented. The objects MPIRT_RESOURCE and MPIRT_TRANSIT
were reintroduced. A general discussion on resource allocation,
resource reservation, admission test, and transition between
real-time mode took place.

It was decided that for now we should restrict this proposal
to MPIRT_GROUP_WORLD and only after that is completed consider other groups.
This decision was based on the following question:
How do an implementation deal with transition with QoS if it
does not know when objects are in use by other groups?

The agreement was reached that there is at most one active
real-time mode per process. This ensures that all objects are
available for transition.

Trade-offs of creating resource objects without knowing future
transitions vs. knowing the entire graph of all transitions
were considered and discussed. It was pointed out that the same
object may be created by two different resource objects
differently, that may not allow transition between them
preserving the state of the object or may not allow transition
with the requested transition QoS. Some discussion concentrated
on an “incremental” admission.

The QoS specification for real-time mode changes was discussed.
It was agreed that initially we should consider only best effort.
The deadlines and events that trigger transition should be
discussed after that. A synchronization of transition from one
real-time mode to another on different processes (processors, nodes)
was considered as part of this QoS discussion.

The topics of the relationship of MPI/RT to the OS, hardware,
and other parts of the system were also partially discussed.

We agreed that there is a need to support some objects to remain
operational during a transition (background channels). The effect
of this to support layered library was also considered.

Arkady agreed to write a proposal that allows to build the
two discussed objects in total isolation, with partial transition
graph, and with the full transition graph. It is up to the user to decide
which information to provide for the creation of these objects.
This proposal will also support operating objects during transition.
A concern was raised to the lack of examples in the document.
A tutorial will be presented in MPIDC’99 and will be available
on MPI/RT web after words (mid March). The example chapter will be
updated and be included in the final document of MPI/RT-1.0.
Half channels (topic 24) was discussed. The following issues
were brought forward:
a. periodic half channels,
b. data streams,
c. interrupts to pull data to half channel buffer,
d. aperiodic interrupts,
e. event kinds for half channels (only receptors), with triggers
as I/O interrupts,
f. interface consistency with existing channel end points specification:
buffers, bufiters, QoS and so on,
g. admission test and QoS specification for half channels,
h. devices (MPI/RT does not control devices, drivers are outside of MPI/RT),
i. data can be “dumped” into a buffer without interrupting CPU,
j. device registration,
k. hooking up half channels with devices,
l. scheduled pulling,
m. random vs. deterministic half channels (dataspecs, data size)

A device driver model with defined “operation-func” that will be invoked
on an interrupt to pull the half channel data into a buffer was accepted.
A completion notification of half channel message transfer uses the existing
channel end point functionality of events on a channel and/or bufiter.

A relationship between MPI/RT and other entities on a platform again
was considered. A consensus is forming that MPI/RT internally specifies
constraint on the use of resources (bus, memory, and so on) for other
entities outside of the MPI/RT.

An issue of multiple devices “dumping” data into the same half-channel
was considered.

An issue of extra states for a half channel was brought forward.

A tentative template of the half channel create function was written:
MPIRT_HALF_CHANNEL_CREATE (
activate_function,
deactivate_function,
operation_function,
…
inbufiter,
outbufiter,
hc_qos,
… other standard channel end point parameters …
*hc);

The direction of a half channel was briefly considered as well
as operations on a half channel.

Shane was “volunteered” to summarize the discussion in the form
of the proposal.
K-slack (topic 8) was discussed. The main issue it to allow
an implementation to transfer more than one buffer at a time
to improve bandwidth. For example, an implementation may be able
to save on synchronization (“ack”, “nack”). The current MPI/RT-1.0
specification allows at most one buffer to be in a channel at a time.
A state transition diagram depicts this requirement. Some protocols
work more efficiently if multiple buffers can be transferred “together”,
for example “sliding window” protocol.

It was agree that “k” means that at most k buffers can be in possession
of a channel at any one time. But there is no requirement or a guarantee
that an implementation will take advantage of that feature.

There was a discussion where we should specify “k”. Early vs. late binding.
It was agreed that the best place to specify it at the early binding,
either as the QoS or as a channel parameter. The forum was leaning to the
later. Channel end points do not require to specify the same slack number.
At commit time the implementation will negotiate the slack number.
It should probably return the negotiated number back to the users.

The biggest discussion centered on new and modified operations on a channel
to support k-slack. The following four operations are under consideration:
a. Wait_All - that returns when all “outstanding” transfers complete
(a “window” transfer is completed),
b. Wait - returns as soon as the “first” outstanding transfer completes
(Notice that several of transfers may complete at the same time).
Other interpretations may also be possible and should be discussed further.
c. Get_Slack - returns the number of outstanding transfers in the channel,
d. Wait_on_specific_buffer - returns when the specified buffer is in the
out-bufiter of the channel. This operation was discussed in great
details on the last day of the meeting. It may require to “poll” on
every buffer of the channel and on every channel that may share
that buffer (or bufiter). Hence, this operation may be quite expensive
and may adversely effect the performance of other objects. We may consider
a different operation that will allow user to find out when the n-th transfer
had completed over the channel.
Once the proposal is written we will return on the issue of the purpose
of these operations and what users are trying to achieve with them.
The relationship between individual Waits and Starts need to be considered
further as well as the changes needed for the channel state transition diagram.

Shane again was volunteered to write a proposal as a representative of MSU.
Unreliable channels (topic 23) were discussed. It was clear that the restriction
that MPI/RT “guarantees that the underlying transmission of messages
is reliable,” that MPI/RT inherited from MPI, need to be reconsidered.
[It was pointed out that this corner stone assumption seems to be missing
from MPI/RT-1.0 document.] After long discussion it became clear that
the forum does not have a clear picture how this unwritten requirement of
the “guarantee delivery” is interpreted. For example, can an implementation
“mask” a member of a group that is not “reliable” so as far as all other
members of the group “active”? Allowing that together with “unreliable” channels
will allow other members of a group to “communicate with it unreliably”.
The dividing line between this topic and fault-tolerance, fault-handling,
dynamic process management, instrumentation and other topics is blur.

Consensus seems to be forming that unreliability should be part of the
channel QoS specification. If a channel is unreliable the requirement
for the implementation to raise an error when the QoS is missed is removed.
The implementation can raise that error but users
should not rely on that for unreliable channels. [This brought a question
on the QoS error for the best effort QoS for the current specification.
Also a timing issue when a buffer is moved from in-bufiter to a channel
and from the channel to the out-bufiter for best effort QoS.]
[It was pointed out that the definition of reliability need to be added to
the glossary.]

The issue of the “global” state view was brought up again.

Arkady and Leonard agreed to flush out more of these issues and summarized
the unreliable channel discussion into a proposal.
The data descriptors (topic 11) were discussed. It was pointed out that strided
DMAs may not improve performance vs. local rearrangement of data in memory and
then transferring it. This is true for DRAM but not for SRAM. It was decided
that we will not now consider strides as part of dataspecs but instead let
users do packing and unpacking themselves. The issue of different dataspecs
in one buffer was considered as well as contiguous memory buffers vs.
scatter-gather ones. It was decided not to address these issues for now but
instead consider two other issues: mutable buffers (late binding of buffer
address) and variable length buffers. Some of the postponed issues will
be reconsidered when we address topic 1 of Data Reorganization.

The definition of structures was considered as part of the above discussion.
The buffer descriptor consists of three parts: dataspec, size, and address.
The first two parts of this 3-tuple is the data descriptor.
The forum agreed to defines a message buffer which is an array of 3-tuples.
This array along with its size will replace a singleton tuple which is
currently used for buffer specification. Each tuple describes a contiguous
memory. A system allocated tuples must be pairwise disjoint for the same
message buffer, but user allocated tuples can overlap. The data descriptor
parts of all tuples are still required to be matched between channel end points.

Steve graciously agreed to write this proposal.
The topic of requiring data descriptors to be the same for channel end points
(topic 22) was briefly discussed. It was agreed that dataspecs must be matched
but that we should relax this requirement for the size. [This discussion
was done independently from the previous one. It was agreed that we will consider
integration of the proposals only after each one of them is approved
independently.]
This flexibility allows users to use the same buffer in multiple channels
instead of creating multiple buffers over the same memory to be used for each
channel. This is part of the early binding specification, hence the implementation
can still set up its channel transfers of the appropriate size during
commit. Details of allocated size and transmitted size will be flushed out
in the proposal. This will not effect the buffer specification but allow
the bufiters for the channel end points to specify different length for
message transmission.

Shane will prepare this proposal.
The issue of SLEEP, WAKE-UP, WAIT-ON (topic 15) was reintroduced. The old
problem of what the “main” application should do while the application is
running as 0-sided communication with handlers to schedule computation?
The old idea of IDLE was reintroduced. The need of blocking on something
was reiterated. The waiting on nothing (IDLE), for time (absolute or
relative time_spec), and on event were considered. It was agreed that a
single operation MPIRT_RECEPTOR_WAIT (receptor, time_spec) will allow to
support all three cases. This operation will be blocking and late binding but the
“special type of a receptor” will be registered (somehow) at the commit time.
Time_spec allows to set up a timeout so that operation will wait on both
time and event. If both specified as NULL then this is equivalent to IDLE.

Dennis volunteered to write a proposal.
The issue of mutable buffers (topic 43) was considered for the
first time. [Again the forum decided to consider this is isolation for
now and not as part of bullets 8, 9 or 12.] It was decided that an operation
for buffer create will have a flag which indicates that the address is
mutable at run time. The default value of the flag will be set to
not mutable to allow backward compatibility. [A more generic version will
be considered when we will merge proposals for this topic with the next
one (topic 44).] The effect of user allocated vs. system allocated memory
as well as bufiter policies that match buffers for channel endpoints will
be considered as part of this proposal.

Steve agreed to write this proposal.
The related topic (topic 44) of flexible buffer size was also discussed
for the first time. It was agreed that we will need two sizes:
maximum possible buffer size and current buffer size. The interpretation
of the maximum size for sending vs. receiving end point of the channel
was discussed. The effect of the push vs. pull model let to the following
agreement. The buffer size must be greater than
the message transmission size specified by the bufiter (see bullet 9).
The backward compatibility need to be addressed as well as the merger
with previous topic.

Steve agreed to write this proposal also.
The issue of multiple running versions of MPI/RT was brought up (topic 46).
The idea of having a “unique” ID for each running version/session was considered.
The sticky issue is the uniqueness of the ID across reboots.
The use of time_spec along with platform specific and MPI/RT specific identifiers
was considered. There was no requirement to get a quick solution. Rather
a need of to consider this ID for debugging purposes, fault identification,
and instrumentation for the future. Consensus was froming to treat it initially
as part of the instrumentation as an extension.

Leonard and Arkady agreed to work on this proposal.
The issue of the channel state transition diagram was revisited. While
the desire was to consider the topic of waiting on an activated channel,
it quickly became apparent that the current figure in MPI/RT-1.0 need some
work. The meaning of labels on the transition, as well relationship
of the condition raising to the states and transitions need to be clarified.
It was agreed that while we will preserve the states that are visible
to the users we will introduce some internal sub-states to them to help
to understand the relationship between condition raisings and the
states. The full description of the figure will be added to the text.
The figure does not represents the requirements but just a tool to help
visualize the states, transitions, and possible generate events that a channel
can do.

Arkady will update the document ASAP and distribute for further comments.

The next meeting of the forum will take place in MITRE in Bedford (Boston)

in April 7-8. It will be 2 days (may be a day and a half). Arkady will host it.

The meeting adjourn at 10:30am on Friday Feb. 5.

The combined list of topics with their current schedules is stated below.

            TOPIC                           |  Expected  |            |
    and person(s) in charge                     | Completion | Discussion |

1 Data-reorganization support | M | E |
Tony, James (from DATA REORG EFFORT) | | |
| | |
2 Collective operations (ALL_TO_ALL) | E | E |
Games, Arkady | | |
| | |
3 Dynamic Process Management, | E-M | E |
Parallel Client Server, etc. | | |
Tony | | |
| | |
4 Interoperation of MPI/RT Implementations | L | E |
Tony | | |
| | |
5 Support for Java binding of MPI/RT | L | M |
(other languages too?) | | |
Tony | | |
| | |
6 Full-QOS-based MPI/RT with mode changes | M | 2/3/99 |
(such as QOS capability for mode changes) | | |
Arkady, Steve | | |
| | |
7 Explicit ability to work with real-time | L | M |
schedulers for process scheduling | | |
Manoj | | |
| | |
8 Work to support k-slack channel mode | M | 2/3/99 |
(whereas MPI/RT 1.0 is 1-slack) | | |
Shane, Tony | | |
| | |
9 Support for IDL for MPI/RT | L | L |
Tony | | |
| | |
10 Fine grain, Coarse grain QOS specifications | M | M |
Arkady, Leonard | | |
| | |
11 More powerful data descriptor to allow | M | 2/3/99 |
strided and non contiguous buffers | | |
James, Tony | | |
| | |
12 how to handle other resources’ schedulers. | L | M |
Currently we have dealt with memory and | | |
network in two different ways and via | | |
completely different specs… | | |
(related to 7 and 10) | | |
Arkady, Leonard, Manoj | | |
| | |
13 Operations on containers. The old issues | E | E |
of propagating the operations to every object| | |
in a container (probably for the containers | | |
that contain homogeneous objects only). | | |
Nathan, Dennis | | |
| | |
14 Container Iterators | E | E |
Nathan, Dennis | | |
| | |
15 MPIRT_SLEEP, MPIRT_WAIT, MPIRT_WAKEUP | E-M | 2/3/99 |
Dennis, Arkady, Tony | | |
| | |
16 Channel state transition (old busy state) | E-M | E |
Cui, Arkady | | |
| | |
17 MPIRT_CHANNEL_WAIT, MPIRT_CHANNEL_TEST | E-M | E |
(related to 16) | | |
Arkady, Cui | | |
| | |
18 QoS Spec for channel (remove restrictions | E-M | E |
that were made for MPI/RT-1.0 at the last | | |
meeting) | | |
Tony, Leonard, Manoj | | |
| | |
19 Default Constructors | E | E |
Cui, Nathan, Shane | | |
| | |
20 New Instrumentation metrics | E-L | E-L |
(Add as needed with experience) | | |
| | |
21 More error returns | E-L | E-L |
Arkady, Tony | | |
| | |
22 Remove restriction of homogeneity on | E-M | 2/4/99 |
bufiter parameters: | | |
dataspec and bufsize for channel endpoints | | |
Shane, Arkady | | |
| | |
23 Unreliable Channels | M-L | 2/4/99 |
Leonard, Tony | | |
| | |
24 Fault Tolerance, Fault Handling, Fault | L | M |
Recovery | | |
| | |
25 Light-Weight Handlers | E | E |
Tony | | |
| | |
36 Half Channels (I/O) | M | 2/3/99 |
Shane, Tony | | |
| | |
37 Default Handlers | E | E |
Tony | | |
| | |
38 Group parameter scope for the commit | M | E |
Tony, Arkady | | |
| | |
39 Mechanisms for passing in externally build | M-L | M |
schedules to MPI/RT | | |
Arkady, Tony | | |
| | |
40 Logical grouping of event kinds | L | M |
(arithmetic on events) | | |
Arkady, Leonard | | |
| | |
41 PGCOs | M-L | M |
Arkady | | |
| | |
42 QoS for handlers (add deadlines) | M | E |
Arkady, Steve | | |
| | |
43 Mutable buffers (late binding of address) | E | 2/4/99 |
Steve, Robert Ginn | | |
| | |
44 Variable length buffers | E | 2/4/99 |
Steve, Robert Ginn | | |
| | |
45 Periodic triggers | M | E |
Steve, Leonard, Arkady | | |
| | |
46 Unique ID for running MPI/RT version | M-L | 2/4/99 |
Leonard, Arkady | | |
| | |
47 QoS for QoS (granularity for raising errors) | L | 2/5/99 |
Leonard | | |

MPI/RT Forum Minutes

Minutes for the Dec. 2-4 2022 MPI/RT Forum meeting held at MITRE in Bedford, MA.

Attendees:
Anthony Skjellum (chair) MSU tony@cs.msstate.edu
Arkady Kanevsky (chair) MITRE arkady@mitre.org
Yogi Dandass MSU yogi@erc.msstate.edu
Shane Hebert MSU shane@erc.msstate.edu
Steve Paavola SKY paavola@sky.com
Manoj Apte MSU manoj@erc.msstate.edu
Zhenquian Cui MSU cuizq@cs.msstate.edu
Dennis Cottell SPAWAR dennis@spawar.navy.mil
James Lebak MIT/LL jlebak@ll.mit.edu
Leonard Monk CSPI lmonk@cspi.com
Nathan Doss LM-GES nathan.e.doss@lmco.com
Anna Rounbehler Raytheon anna_c_rounbehler@res.raytheon.com
Sharon O’Neal Raytheon shoneal@west.raytheon.com
Richard Games MITRE rg@mitre.org

Summary of official votes:

Objects chapter: third and final formal vote - 9/0/2
Types and Datatypes Chapter: third and final formal vote - 9/0/2
Handler chapter: third and final formal vote - 8/0/3
Event chapter: third and final formal vote - 7/0/5
Buffer Management chapter: third and final formal vote - 10/0/2
Channel overview chapter: third and final formal vote - 8/0/3
Point to point channel: third and final formal vote - 9/0/2
Collective Channel chapter: third and final formal vote - 9/0/2
Channel Operations: third and final formal vote - 7/1/3
Quality of service specification chapter: third and final
formal vote - 7/0/3
Committing Objects chapter: third and final official vote - 8/0/2
QoS Overview Chapter: third and final formal vote - 6/0/3
Instrumentation Chapter: third and final formal vote - 6/1/3
The meeting came to order at 8:30 AM on December 2, 2022
An etiquette of email discussion was briefly discussed.
We decided to include “MPI/RT” on the subject line.
Also we decided to reply only to the entire mailing list to
reduce message duplication and to ensure that the whole discussion
appears on the reflector.
Institutions represented at the meeting: LMCO/GES,
MSU, MSTI, Sky Computers, SPAWAR, MIT/LL, MITRE, CSPI, and Raytheon.
A brief walk through of the small changes to
the document posted for public comments was performed.
The proposals that were discussed on the MPI/RT exploder were officially
presented. Since less than half of the eligible institutions voted
electronically on some of proposals and no official votes were taken
electronically on others, the official votes were taken again on all
the proposals.

A. Proposal 1: Container Iterators. There was a general agreement that
the proposal is good but it needed more details, especially for
“container_iterator”. It was unanimously decided to postpone inclusion
of this functionality until MPI/RT-1.1. Nathan Doss will provided an
updated proposal on the exploder (see MPI/RT-1.1 below).

B. Proposal 2: Buffer Iterators. This is the final resolution of the two
initial proposals that deal with performance issues for message
transfers and error checking. The following amendments were added:

  a. High performance implementations may choose to skip error
         detection code.  Implementations can offer a variety of
         optimization options to assist users in the debugging process.
    
      b. Added advice to users and implementers to reflect the first
         amendment.  An analogous statement should also go into the
         introduction chapter (chapter 1).
    
      The official vote for inclusion of this proposal as part of MPI/RT-1.0
      document passed 8/0/0. Chapters 5, 6, and 9 were updated with the
      proposal and presented on the second day of the Forum (Dec. 3) for the
      official (final) votes on the chapters.

C. Proposal 3: MPIRT_SLEEP. It was unanimously decided that this needed
further discussion and should be postponed until MPI/RT-1.1. For
MPI/RT-1.0
it was decided to add advice to users to use system dependent calls
for
this functionality.

D. Proposal 4: 2-sided message transfers. There was a long discussion
centered around the states of the channel objects and the effect of
operations on the channel (start, activate, deactivate) and the state
of
the channel when a message transfer completes. As a result, the text
of
the proposal, and the state transition diagram for channels was
modified.
“Deactivate” on a the started channel is now disallowed. “Global
completion” and “global start” are no longer part of MPI/RT-1.0. Some
terminology related to the channel states was changed.

  The official vote on the modified proposal to be part of MPI/RT-1.0
      passed 7/0/0.
    
      Chapters 6 and 9 were updated with the approved proposal and presented
      on the second day of the Forum (Dec. 3) for official (final) votes.

E. The latex error in chapter 8 that forced the rest of the document to
be in italic was noted and fixed for the final official votes taken on
the second day of the Forum.

Remove “Draft” from the title page.
Preface: No formal vote required. The following changes were agreed
upon:
a. Remove footnote.
b. Add paragraph for thanks to institutions that hosted the Meetings:
MITRE, MSU, SPAWAR, and SANDIA.
Minor changes were made to the Glossary.
Introduction (chapter 1): final formal vote - 7/0/0. Amendments:
A. Add a section 1.4 “What to expect in implementations”
a. Two implementation libraries: high-performance and “debug” mode.
b. Errors may not be detected in high-performance implementation.
c. Erroneous programs produce undefined results.
B. Move all the references to MPI into footnotes so the reader does
not have to know MPI to read this document.
C. Add section on the sidedness of message transfers.
D. Add other advices.
MPI/RT Objects (Chapter 2): final formal vote - 7/0/0. Amendments:
A. Do not add default constructors in MPI/RT-1.0. Write a proposal
for MPI/RT-1.1.
B. Change MPIRT_OBJECT_COMPARE to MPIRT_OBJECT_IS_EQUAL which has
only has only two returns MPIRT_EQUAL and MPIRT_NOT_EQUAL.
[This change effects every object in the document and will be done
globally for all chapters.]
Write a detailed proposal for an expanded compare function for
MPI/RT-1.1. Consider a special compare function for containers.
C. Fix section headings for generic container and vector accessor
functions
in section 2.4.1.
D. Add MPIRT_GROUP_SELF in section 2.5.
Types and Dataspecs (Chapter 3): final formal vote - 7/0/0.
Only minor editorial comments, a clearer definition of transmittable
MPI/RT types, and the integer values for MPIRT_TRUE and MPIRT_FALSE.
Event Delivery Abstraction and Handlers (Chapter 4): final formal
vote - 5/0/1. Amendment:
A. Synchronize the use of “kind of” event with “class of” events
throughout
the document. Add the chosen term to the glossary.
Buffer Management (Chapter 5 with the proposal 2 - see above):
final formal vote - 9/0/0. Amendments:
A. Remove “semi-opaque” from the figures.
B. Add missing get/set operation for the new bufiter parameters.
C. Add the following rules for a buffer (resolution of prolong discussion
on when it is safe to read/write into memory of a buffer):
a. A buffer object is available if it is committed, until it is
inserted
(into a bufiter) and after it is removed from a bufiter, until it is
inserted again.
b. It is illegal to read/write to a location after commit,
unless every committed buffer object whose memory region contains
that location is available.
c. It is illegal to insert a buffer unless it is available.
d. It is illegal for a buffer object to be in the initial buffer list
of more than one bufiter being committed.
e. The implementation will not read/write a location unless
at least one user-created buffer containing that location is
inserted
into a bufiter.
D. Add a buffer state transition diagram (which was agreed upon at the
meeting).
E. Come up with the better name for operations in section 5.3.
a. Replace MPIRT_BUFFER_CREATE_SUBSET with
MPIRT_BUFFER_CREATE_SUBBUFFER.
F. Fix C++ binding for MPIRT_CVECTOR_CREATE_BUFFERS.
G. Fix C++ binding for MPIRT_BUFITER_CREATE.
H. Synchronize the consistent use of formats for S- and V-containers.
I. Fix Bufiter state transition diagram.
J. Fix old get/set operations for bufiter parameters to use new terms.
Channel overview (Chapter 6 with the proposals 2 and 4 - see above):
final formal vote - 9/0/0. Amendments:
A. If MPIRT_ACCEPT_ANY is used for all channel endpoints then it is
equivalent to MPIRT_BEST_EFFORT for all endpoints.
B. Many editorial changes.
Point-To-Point Channels (Chapter 7): final formal vote - 8/0/0.
No amendments.
Collective Channels (Chapter 8): final formal vote - 8/0/0.
Amendments:
A. Remove “Shadow” from all advices to implementers.
B. Change “implementation” to “example” in all advice to implementers.
C. Fix latex italic problem.
D. Editorial changes.
Channel operations (Chapter 9 with the proposals 2 and 4 - see above):
final formal vote - 8/0/0. Amendments:
A. Change “Deactivated” state to “IDLE” state.
B. It is illegal to “wait” on activated channel (for MPI/RT-1.0).
(We will study this issue for MPI/RT-1.1).
C. “Wait” on an idle channel returns immediately, without error.
D. Channel should be an IN parameter for MPIRT_CHANNEL_WAIT and TEST
operations.
E. Editorial changes.
QoS Overview (Chapter 10): final formal vote - 8/0/0.
A lot of editorial changes.
QoS Specification (Chapter 11): final formal vote - 6/0/3.
Amendments:
A. For Time driven channels:
a. MPIRT_TIME_SPEC_IGNORE is not allowed as a start parameter.
b. If start and deadline are specified as absolute, the time period
should be MPIRT_TIME_SPEC_IGNORE.
In this case only one message transfer will take place.
c. If period is specified (not IGNORE nor NULL) then start and deadline
must be relative.
d. Rename “start” to “phase.”
e. Phase is less than or equal to the period.
B. For event-driven channels:
a. If the QoS spec is the same for all channel endpoints then the
occurrence of the event transfers only one message over the channel.
C. Remove MPIRT_QOS_IGNORE.
D. More detailed, and less restrictive specifications will be considered
for MPI/RT-1.1.
E. Editorial changes.
Committing Objects and Resource Allocation (Chapter 12) - final formal
vote 9/0/0. Amendments:
A. It is the user’s responsibility to ensure that there are
no operations pending when the commit function is called.
B. The objects that were committed before the commit() call
retain their states across the commit() call.
C. Copy Figure 13.1 into this chapter.
Initialize and Termination (Chapter 13) - final formal vote 9/0/0.
No amendments.
Clocks (Chapter 14) - final formal vote 9/0/0.
No amendments.
Instrumentation (Chapter 15) - - final formal vote 8/0/1.
Amendment:
A. Change MPIRT_SYSTEM to MPIRT_OBJECT_NULL.
The final vote on the entire document as amended - 8/0/1.
Reinsert the Function Names index.
Add a section on “How to read the document” describing typographical
conventions.
Fix all C++ bindings.
Resize figures to fit.
Remove the references to MPI/RT-1.0 and MPI/RT-1.1 from the document.
Change all the derivatives of the MPIRT_OBJECT_COMPARE function to
reflect
the change to MPIRT_OBJECT_IS_EQUAL.
The last day of the Forum was spent on discussion of MPI/RT-1.1 and
beyond. It was agreed that for MPI/RT-1.1 we should just make small
incremental additions to MPI/RT-1.0. As a topic is resolved (similar
to the two proposals that where voted in during this meeting) and passes
three formal votes, it will become a separate chapter of MPI/RT-1.1 and
will be merged with MPI/RT-1.0.

Below is the list of topics we are considering. For each topic we asked
two
questions:
A. What is the expected time for completion?
B. How quickly should we start discussing it?
The possible answers are: E - early, L - later, and M - middle.
E - answer for the first question means that it is part of MPI/RT-1.1.
L - answer for the first question means that it is not part of
MPI/RT-1.1.
M - answer for the first question means that we may have at least a
part
of the topic resolved for MPI/RT-1.1.

Here is the list of the topics discussed, the answers to the two
questions,
and the person(s) in charge of the topic to ensure its progress.

TOPIC | Expected | Expected |
and person(s) in charge | Discussion | Completion|

1 Data-reorganization support | M | E |
Tony, James (from DATA REORG EFFORT) | | |
| | |
2 Collective operations (ALL_TO_ALL) | E | E |
Games, Arkady | | |
| | |
3 Dynamic Process Management, | E-M | E |
Parallel Client Server, etc. | | |
Tony | | |
| | |
4 Interoperation of MPI/RT Implementations | L | E |
Tony | | |
| | |
5 Support for Java binding of MPI/RT | L | M |
(other languages too?) | | |
Tony | | |
| | |
6 Full-QOS-based MPI/RT with mode changes | M | E |
(such as QOS capability for mode changes) | | |
Arkady, Steve | | |
| | |
7 Explicit ability to work with real-time | L | M |
schedulers for process scheduling | | |
Manoj | | |
| | |
8 Work to support k-slack channel mode | M | E |
(whereas MPI/RT 1.0 is 1-slack) | | |
Tony | | |
| | |
9 Support for IDL for MPI/RT | L | L |
Tony | | |
| | |
10 Fine grain, Coarse grain QOS specifications | M | M |
Arkady, Leonard | | |
| | |
11 More powerful data descriptor to allow | M | E |
strided and non contiguous buffers | | |
James, Tony | | |
| | |
12 how to handle other resources’ schedulers. | L | M |
Currently we have dealt with memory and | | |
network in two different ways and via | | |
completely different specs… | | |
(related to 7 and 10) | | |
Arkady, Leonard, Manoj | | |
| | |
13 Operations on containers. The old issues | E | E |
of propagating the operations to every object | | |
in a container (probably for the containers | | |
that contain homogeneous objects only). | | |
Nathan, Dennis | | |
| | |
14 Container Iterators | E | E |
Nathan, Dennis | | |
| | |
15 MPIRT_SLEEP, MPIRT_WAIT, MPIRT_WAKEUP | E-M | E |
Arkady, Tony | | |
| | |
16 Channel state transition (old busy state) | E-M | E |
Cui, Arkady | | |
| | |
17 MPIRT_CHANNEL_WAIT, MPIRT_CHANNEL_TEST | E-M | E |
(related to 16) | | |
Arkady, Cui | | |
| | |
18 QoS Spec for channel (remove restrictions | E-M | E |
that were made for MPI/RT-1.0 at the last | | |
meeting) | | |
Tony, Leonard, Manoj | | |
| | |
19 Default Constructors | E | E |
Cui, Nathan, Shane | | |
| | |
20 New Instrumentation metrics | E-L | E-L |
(Add as needed with experience) | | |
| | |
21 More error returns | E-L | E-L |
Arkady, Tony | | |
| | |
22 Remove restriction of homogeneity on | E-M | E-M |
bufiter parameters: | | |
dataspec and bufsize for channel endpoints | | |
Shane, Arkady | | |
| | |
23 Unreliable Channels | M-L | E |
Leonard, Tony | | |
| | |
24 Fault Tolerance, Fault Handling, Fault | L | M |
Recovery | | |

Minutes for the 22 July 2022 MPI/RT Forum meeting held in Washington DC.

Attendees:
Anthony Skjellum (chair) MSU tony@cs.msstate.edu
Arkady Kanevsky (chair) MITRE arkady@mitre.org
Jon Hiller STA jhiller@stassociates.com
Joe Fogler UNM/AHPCC fogler@arc.unm.edu
Yogi Dandass MSU yogi@erc.msstate.edu
Shane Hebert MSU shane@erc.msstate.edu
Steve Paavola SKY paavola@sky.com
Manoj Apte MSU manoj@erc.msstate.edu
Zhenquian Cui MSU cuizq@cs.msstate.edu
Dennis Cottell SPAWAR dennis@spawar.navy.mil
Randall Judd SPAWAR/TASP judd@spawar.navy.mil
David Coker ISI coker@sensors.com
James Lebak MIT/LL jlebak@ll.mit.edu
Michael Greico ASEC mjgrieco@jswg.org
Leonard Monk CSPI lmonk@cspi.com
Nathan Doss LM-GES nathan.e.doss@lmco.com
Anna Rounbehler Raytheon anna_c_rounbehler@res.raytheon.com
Robert Babb DU babb@cs.du.edu
Chris Hayden Sanders christopher.p.hayden@lmco.com

Formal voting institutions:
MSU, MITRE, UNM/AHPCC, SKY, SPAWAR, SPAWAR/TASP, MIT/LL, ASEC, LM-GES,
Raytheon, DU, CSPI, Sanders

Summary of official votes:

Objects chapter: second formal vote - 9/0/2
Types and Datatypes Chapter: second formal vote - 9/0/2
Handler chapter: second formal vote - 8/0/3
Event chapter: second formal vote - 7/0/5
Buffer Management chapter: second formal vote - 10/0/2
Channel overview chapter: second formal vote - 8/0/3
Point to point channel: second formal vote - 9/0/2
Collective Channel chapter: second formal vote - 9/0/2
Channel Operations: second formal vote - 7/1/3
Quality of service specification chapter: second formal vote - 7/0/3
Committing Objects chapter: second official vote - 8/0/2
QoS Overview Chapter: second formal vote - 6/0/3
Instrumentation Chapter: second formal vote - 6/1/3
The forum thanked Jon Hiller and STA for hosting the forum meeting in
Washington DC.
June minutes passed - 11/0/3
Glossary:
a. Identify those objects that are defined by MPI/RT.
b. Look through document for other undefined terms.
Introduction
a. Add a high level description of how the user sets up an MPI/RT
program.
b. Add an introduction for implementers.
Objects chapter: second official vote - 9/0/2
a. Add advice to user that objects they create should be freed.
b. Explain MPIRT_OBJECT_TYPE
c. Explain that there are MPI/RT types that are directly accessible
by the user, as opposed to opaque MPI/RT objects, and a subset
of these are transmittable.
d. Rewrite overview section to be clearer.
e. Several fixes to object figure - Add individual collective
channels. Make figure landscape and larger.
f. Object body description - add diagram to make clearer.
g. Remove - section 2.1.1 header and possibly move to chapter 7
just before QoS issues. Consider precis for chapter 1.
h. All objects have names. However, certain objects do not need
unique names, and can use NULL for the object names. Write
description that commit can fail when the objects requiring
unique names do not have unique names.
i. The Object name does not have an impact on the compare function.
j. Remove references to “caching.”
k. Make “Attributes are scalar…” advice to users be part of
regular text.
l. Vector containers - remove Cvector_int. We now have a
CVector_base object with descendants: CVector and Groups.
Types and Datatypes Chapter: second official vote - 9/0/2
a. Basic Datatypes are descriptors, not declarators.
b. MPIRT_datatype are now objects. Added to hierarchy diagram.
c. DATATYPE is now called DATASPEC.
d. Need DATATYPE_BOOL, DATATYPE_TIME, DATATYPE_INT64
e. Type: Need MPIRT_INT64, with associated arithmetic operators.
f. Mod for Int64 follows C/C++ rules.
g. Move section 3.3 to section 3.1
h. Remove DATASPEC_COMPLEX
i. Remove DATASPEC_LONG_DOUBLE unless it is in the ANSI C standard.
j. Time spec types are MPIRT_TIME_ABSOLUTE, MPIRT_TIME_IGNORE, and
MPIRT_TIME_RELATIVE.
Handler chapter: Second Formal vote - 8/0/3
a. Merge events and Handler chapters. Event chapter first.
b. Make all tables have smaller fonts so they fit on the page,
perhaps using vertical struts.
c. Event Objects should now be called Control Channel/Control
Network.
Event chapter: Second formal vote - 7/0/5
a. Event class renamed to EvDel (Event Delivery System).
b. Need a figure for event delivery abstraction
c. Add exceptions for error codes in C++ bindings.
d. Three proposals were considered.
i. - 1: There is only 1 explicit receptor for event per
process.
2: The retrieve receptor retrieves the explicit
receptor.
3: QoS Chapter - Add group parameter to PrTiEv, PrEv, and
Constrain object.
ii. - 1: Receptors have only async handlers
2: Multiple receptors per event name per process allowed.
3: Objects can have async handlers (callbacks).
4: Silent on implementation of “implicit receptors.”
5: Extend Qos parameter for “implicit receptor.”
iii. - 1: Multiple receptors
2: Silent on order for async handlers.
3: No retrieve function
The following were voted on:
A. proposal iii - 4 votes.
B. proposal i with points 1 and 2 only - 4 votes.
C. proposal i with point 1 and proposal iii with
point 3 - 4 votes.
e. The previous proposals were clarified by the following votes:
i. Single explicit receptor per event name per process - 10/6/0.
ii. Retrieve function only returns explicit receptors - 13/1/2.
Buffer Management chapter: second formal vote - 10/0/2
a. Create picture for MPIRT_CVECTOR_CREATE_BUFFERS to clarify.
b. Figure: need to add transitions to/from uncommitted bufiter to
partially full and full states.
Channel overview chapter: second vote - 8/0/3
a. Add “Introduction” section heading where missing (all
chapters).
b. Add QoS failure situations to state transition diagram.
c. When the failure happens during a buffer transition, the
implementation retries as long as QoS permits. When QoS failure
happens, the buffer is moved to out-bufiter and the event is
raised. Write an example demonstrating this situation.
d. Add QoS BestEffort when channel doesn’t need QoS. Add text in
the QoS Chapter.
e. Make sure all channel conditions appear in this chapter.
Point to point channel: second formal vote 9/0/2
Collective Channel chapter: second formal vote: 9/0/2
a. Fix figure 9.1
b. Write short descriptions of what each operation means.
Channel Operations: Formal vote - 7/1/3
a. Fix figure 10.1
b. Channel_Test() returns MPIRT_CHANNEL_BUSY and MPIRT_CHANNEL_IDLE.
c. Proposal: Channel state transition has 2 states idle and running.
Two conditions channel_started and channel_completed.
Passed - 9/3/3
d. Proposal to delay formal vote until text is modified to reflect
the changes in the chapter was defeated - 3/9/0.
Examples moved to appendix.
Quality of service specification chapter: Second official vote - 7/0/3
a. The channel QoS specification for QOS_CHANNEL_PREV and
QOS_CHANNEL_PRTIEV lacks the QoS sepcification for the implicit
receptors. In order to correct this, the following functions will be
added to start_event for QOS_CHANNEL_PREV and QOS_CHANNEL_PRTIEV:
start_scope(group), and start_qos(qos_receptor). For stop_event
for QOS_CHANNEL_PREV the stop_scope(group) and stop_qos(qos_receptor)
will be added. The corresponding accessor operations will also be
added.
Committing Objects chapter: second official vote - 8/0/2
a. Commit - Each object is committed once.
b. Explicitly state that all objects do not need to be explicitly listed
in commit container. Implicit relationships, for example bufiter for
channels, will be committed automagically.
QoS Overview Chapter: second formal - 6/0/3
a. EvDel needs to be reflected in this chapter.
b. Conditions for channels need to be added to this chapter.
Instrumentation Chapter: second Formal vote - 6/1/3
a. Replace MPIRT_String with String_name.
b. Write explanation for minimum set of metrics table.
c. Replace MPIRT_SYSTEM with MPIRT_OBJECT_NULL.
Proposal to keep container operations in version 1.0 defeated - 5/6/2.
Init-term chapter:
a. Friendly amendment - add function and constant to return version number
of MPI/RT.

Draft Minutes for the June 23-25 2022 MPI/RT forum meeting held at Mississippi State University

Attendees:

Anthony Skjellum (chair) MSU tony@cs.msstate.edu
Arkady Kanevsky (chair) MITRE arkady@mitre.org
Darwin Ammala MPI Software Tech dammala@mpi-softtech.com
Steve Paavola Sky paavola@sky.com
Dennis Cotttel SPAWAR dennis@spawar.navy.mil
James Lebak MIT/LL jlebak@ll.mit.edu
Joe Fogler MHPCC/AHPCC fogler@arc.unm.edu
Zhenquian Cui MSU cuizq@cs.msstate.edu
Greg Henley MSU henley@erc.msstate.edu
Nathan Doss Lockheed Martin-GES natahn.e.doss@lmco.com
Marc Fidler Sanders marc.d.fidler@lmco.com
Robert George MSU robert@erc.msstate.edu
George Crawford III MPI Software Tech crawfford@mpi-softtech.com
Yogi Dandass MSU yogi@erc.msstate.edu
Michael Grieco ASEC/NSA mjgrieco@jswg.org
Manoj Apte MSU manoj@erc.msstate.edu
Shane Hebert MSU shane@erc.msstate.edu
Eric Holbrook MPI Software Tech eric@mpi-softtech.com
Clay Taylor, Jr. MPI Software Tech cdtaylor@mpi-softtech.com
Thomas McMahon MPI Software Tech thom@mpi-softtech.com
Mark Lin MPI Software Tech mlin@mpi-softtech.com
Robert Babb DU babb@cs.du.edu
Gabriel Jones MSU gabriel@cs.msstate.edu

Summary of official votes:
Handlers (chapter 4): first official vote - 9/0/1
Event objects (chapter 5): first official vote - 11/0/0
Object Decoration (Section 2.3): first official vote 11/0/0
Containers (section 2.4): first official vote 11/0/0
Groups (section 2.5): official vote 11/0/0
Datatypes (Chapter 3): first official vote 11/0/0
Handler QoS (Section 14.2): first official vote 11/0/0
Event QoS (Section 14.3): first official vote 9/0/0
Initialization and Termination (Chapter 15): second official vote 9/0/0.
Instrumentation (Chapter 17): first official vote 9/0/2

The meeting came to order at 8:30 AM on June 23, 2022
The minutes from the previous meeting was discussed and approved
after agreeing that the vote recorded in the minutes for the QoS
chapter was limited to the QoS specification for channels.
Approved: 16/0/1
Institutions represented at the meeting for official votes:
Sanders, Lockheed Martin/GES, MSU, MPI Software Tech, Sky, SPAWAR,
ASEC/NSA,MIT/LL, MITRE and MHPCC/AHPCC.
A brief walkthrough of the current document was performed.
Handlers (chapter 4): official vote - 9/0/1
a. If the object that triggered an event is not local to the
handler function (in the same process), the handler function
gets MPIRT_OBJECT_NULL and MPIRT_CONDITION_UNKNOWN in the
object and condition parameters respectively.
b. Handlers may be synchronous or Asynchronous. Added a flag in
the constructor to reflect this. Also add an accessor function
to retrieve/set the flag.
c. First the asynchronous handlers are present to the scheduler,
then the synchronous handlers are started.
d. Move light weight operations that generate upcall handler calls
(section 4.2.2) to MPI/RT 1.1.
e. A recoverable event delivers the results of the synchronous
handlers for the local receptors back to the MPI/RT object that
raised the event trigger indirectly. The implementation uses the
return value from the handler function to determine if a retry of
the operation which caused the recoverable event can be performed.
f. Lightweight handlers are implemented as functions.
Event objects (chapter 5): official vote - 11/0/0
a. The “event” prefix in the LIS specification parameters for
certain functions was removed. For example, event_qos and
event_scope in MPIRT_TRIGGER_CREATE are now qos and scope,
respectively.
b. Users can create triggers with the same names as are used for
the system generated implicit triggers.
c. The proposal to make the MPIRT_TRIGGER_RAISE function
asynchronous approved by straw vote - 20/0/2.
d. The proposal to allow at most one receptor for a given event
name per process (regardless of handler type: synchronous or
asynchronous) was approved by straw vote - 17/0/2.
e. The proposal to add a function to retrieve the receptor
associated with the event name in the local process
was approved by straw vote - 9/4/6.
f. Raising an event is a non-blocking operation.
MPI/RT Objects (Chapter 2)
a. Attribute Caching changed to Object Decoration (Section 2.3) -
official vote 11/0/0
i. Change “copy” in function names and bindings to “dup”.
ii. Remove Callback from the object hierarchy diagram since it
is only a function, not an object.
iii. change DECOR_KEYVAL in LIS to KEYVAL.
iv. Add LIS for functions in the section.
v. Add datatype of “choice” for void * parameters in LSI
declarations.
vi. Clarification was made that duplicating a KEYVAL creates a
new object with same callbacks, but a unique ID.
vii. Added accessor (get/set) functions for KEYVAL callbacks.
viii. Removed the MPIRT_KEYVAL_FREE function because it is
available at the MPIRT_OBJECT level.
b. Container (section 2.4) - official vote 11/0/0
i. Shorten object names to MPIRT_CSET, MPIRT_CVECTOR,
MPIRT_CVECTOR_INT.
ii. Proposal to keep “syntactic sugar” (functions of
convenience, but not essential to the standard) such as
union, concatenate, difference etc was defeated by straw
vote - 0/13/5.
iii. Proposal to keep vector functions incl, excl was rejected
by straw vote - 1/13/4.
iv. Function to compare containers was removed because it
is common to all objects. The comparison function should
be at the MPIRT_OBJECT level.
v. The proposal to keep function TRANSLATE_INDICES was rejected
by straw vote - 0/16/2.
vi. Removed _ELEMENT from LSI bindings for all containers. For
example MPIRT_CVECTOR_INT_REMOVE_ELEMENT becomes
MPIRT_CVECTOR_INT_REMOVE.
vii. Move common functions to the top of the chapter (just as in
other chapters/sections).
c. Groups (section 2.5) - official vote 11/0/0
i. Group containers must perform automatic compaction when
elements (process addresses) are removed, ensuring there
are no “holes” in a group container.
ii. GROUP_WORLD is read-only.
iii. Containers for group world are identical on all processes
(that is, the order of task_addresses is the same in all of them).
d. Miscellaneous Types (section 2.6) official vote: 11/0/0
Datatypes (Chapter 3) - official vote 11/0/0
a. Move section 3.2 to chapter 2 and rename as “Miscellaneous Objects”
Miscellaneous objects are: MPIRT_STRING_NAME, MPIRT_TASK_ADDRESS.
b. MPIRT_STRING object was removed. It is now a typedef to char*.
c. Need to create a table of serializable MPI/RT objects that can
be transmitted.
Quality of Service (Chapter 14)
a. Handler QoS (Section 14.2) - official vote 11/0/0
i. QoS for handlers can only be priority and timeout.
It was decided that QoS should only support objects that are
created/maintained by the implementation. Since handlers are
just user written code, the implementation does not know how
long it takes to execute, nor any hidden dependencies (data
dependencies etc.).
b. Event QoS (Section 14.3) - official vote 9/0/0
Instrumentation (Chapter 17) - official vote 9/0/2
a. Elapsed time in MPI/RT Metrics (Section 17.2) was changed to timer.
b. The concept of attaching a probe to a user object was removed.
c. In MPI/RT User Metrics (Section 17.4) type_size was changed to
data_size, where data_size is the number of data_types in the
accumulator.
d. MPIRT_PROBE_RETRIEVE_DATASIZE function was added to section 17.3.
e. Probes work only on local MPI/RT objects.
Initialization and Termination (Chapter 15) - official vote 9/0/0.
a. Added advice to users that implementations can modify argc and
argv parameters.
b. Minor changes to figure to show that the commit operation, while
the application is in a real-time mode, causes the application to
go into a non-real-time mode until the commit operation succeeds.

DRAFT Minutes of the MPI/RT Forum Meeting of May 19-21, 2022

Name/Organization e-mail
Leonard Monk, CSPI lmonk@cspi.com
James Lebak, MIT/LL jlebak@ll.mit.edu
Manoj Apte, MSU manoj@erc.msstate.edu
Dennis Cottel, SPAWAR dennis@spawar.navy.mil
Zhenquin Cui, MSU cuizq@cs.msstate.edu
Yogi Dandass, MSU yogi@erc.msstate.edu
Nathan Doss, Lockheed-Martin/GES ndoss@motown.lmco.com
Marc Fidler, Sanders marc.d.fidler@lmco.com
Robert Ginn, NAWC, AD rob@sun701.nawcad.navy.mil
Chuck Browne, NAWCAD chuck@sun701.nawcad.navy.mil
Shane Hebert, MSU shane@erc.msstate.edu
Randy Judd, SSC-SD, TASP judd@spawar.navy.mil
Arkady Kanevsky, MITRE arkady@mitre.org [co-chair, host]
Anthony Skjellum, MSU tony@cs.msstate.edu [co-chair]
Clay Taylor, MPI Soft. Tech. cdtaylor@mpi-softtech.com
Steve Paavola, SKY paavola@sky.com
Anna Rounbehler, Raytheon Anna_C_Rounbehler@res.ray.com
Joe Fogler, AHPCC/MHPCC fogler@arc.unm.edu
Chris Hayden, Sanders christopher.p.hayden@lmco.com

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, SKY, MITRE,
LM GES, Raytheon, Sanders, NAWC, MPI Software Technologies

Official vote summary for meeting (chapter numbers from May 18 version of the document:
Formal:

Minutes of the previous meeting 14/0/2.
Event Object (chapter 8) - first vote passed 7/1/0.
Buffer Management (chapter 9) - first vote passed 8/0/1.
Channel Overview (chapter 12) - first vote passed 8/0/0.
Point-to-Point Channels (chapter 13) - first vote passed 7/0/0.
Collective Channels (chapter 14) - first vote passed 7/0/0.
Completable chapter (chapter 11) - first vote passed 7/0/0.
Data Transfer chapter (chapter 15) - first vote passed 7/0/0.
Time Spec section (section 18.3) - first vote passed 9/0/0.
Event-Driven chapter (chapter 19) - first vote passes 7/0/2.
Priority-Driven chapter (chapter 20) - first vote passed 9/0/0.
QoS Specification (chapter 21) - first vote passed 7/1/1.

Straw votes [selected]

Change back to a single commit/uncommit operation (chapter 10) 8/7/0.
Parallel Programming Models and Groups (chapter 5) voted PCOMM proposal down 17/0/0

The meeting came to order at 8:30am on May 19.

The minutes of the previous meeting were approved. Official vote 14/0/2.
The meeting opened with a discussion of the newest draft, which was distributed
at the meeting, and put on the reflector as of May 18. A walk through
of the document was offered, with highlights and lowlights offered.
It was noted that several new chapters have appeared, several
have been moved, while others rewritten and “cleaned up” for reading.
The biggest changes are:
document is split into two books: Standard and Best Engineering Practices,
introduction for users,
container section of MPI/RT objects,
parallel Programming Models replaced group, communicators and context chapter,
datatypes, light-weight handlers and upcall handlers,
event objects, commitable objects, completable objects and QoS specification.
Reading of “Parallel Programming Models and Application Formulation”
chapter #5 (old groups, communicators and context chapter) was undertaken.
– The groups replaces communicators as the scope defining entity.
The primitive communicators were discussed that allows to have
a very restricted MPI-1 type communication for setting up real-time
channels.
a. The PCOMM (primitive communicator) proposal was voted down;
straw vote 17/0/0
The main reason for that is the that allows two different ways
to do message transfers: channel way and send/receive way.
b. The group operations were discussed. Since groups are containers
it was decided that the group operation of this chapter should
be moved into container section of the MPI/RT objects chapter
and generalized for vector containers.
c. There are no non-local operations on the groups.
d. There is only one group specific operation (not a vector container
operation) - GET_RANK.
e. The remainder of this chapter (about one page) became
a section of the MPI/RT object chapter.
Reading of the Event Object chapter was conducted.
The chapter was accepted with official vote of 7/1/0 with the following
changes:
a. There is no need for the handlers for individual objects.
The only one needed is for event receptors. That means that
if users (application or implementation) would like to invoke
a handler on some object condition then can create an event
receptor for that object condition name. (The handler chapter
was not read during this meeting so the effect on up-call handlers
is not clear!)
This change does not effect this chapter in itself but it means
that handlers are removed for all objects except event receptors.
b. Move a paragraph on MPIRT_NAME_NULL into object chapter.
c. Create a separate section in the chapter for common event functionality
(get, set) away from triggers and receptors sections.
d. Put back the disappeared TRIGGER_RAISE operation.
e. Change the names of Receptor operations from GET/SET for FLAG to
IS_ENABLE and IS_DISABLE respectively.
```
    The updated chapter was distributed on May 21.
    
```
Reading of the Buffer Management chapter was conducted.
The chapter was accepted with official vote of 8/0/1 with the following
changes.
a. Remove most of the advice to implementors in the intro section
and move the remainder of it after buffer_create operation.
Delete text prior to buffer_create in buffer object section.
b. Change the bufset section completely. The new functionality
should support buffer partitioning:
Create container of buffers by chopping a “big” buffer
into number of “small” buffers preserving the datatype and by creating
a buffer for the reminder of the “big” buffer.
Create a buffer that starts at specified displacement and of
specified length in datatype items.
All containers have names.
The containers for buffers should be container vectors.
Add advice to implementors on special memory and continuous memory.
c. The buffer iterator section should be modified. Get rid of buflabels
as a separate parameter for the iterator. Add two operations to
assign a label for a buffer and to retrieve a label of a buffer.
All buffers have default labels on commit.
Delete all other operations that deal with labels and remove
labels from buffer iterator accessor operations.
Add text that all buffers in the bufiter are of the same size and
datatype and not necessarily from the same container.
Buffers are not placed into bufiter until commit.
Add advice to implementors and examples.
Fix the figure.
A short discussion of the datatype chapter took place. It was unanimously agreed
to keep basic datatype section plus add C++ mapping and MPI/RT specific datatypes.
Remove the rest of chapter. Split MPI/RT datatypes into ones that are transferred
in a message and the ones that are not. Create a separate non-transmittable datatypes
section for the later.
The Channel overview chapter was read. It was voted in with official vote
of 8/0/0 with the following changes:
a. Move rationale to point-to-point channel chapter.
b Beef up descriptions of channel attributes
c. Add “attributes” to the name of the section for common operations.
d. Add forward reference to point-to-point and collective channels
sections.
e. Synchronize the use of the names with point-to-point and collective
channels chapters.
f. Percolate the changes for event object chapter (dropping handlers)
to this chapter.
The point-to-point channel chapter was read and voted in with official vote of 7/0/0.
a. The main changes deal with a name that is used as a generic
object names, for the matching of channel endpoints and for channel
event names. After long discussion and a lot of work we agreed that
there is only one name used.
For events the name implicitly generated using the following rule
by concatenating parts separated by “:”:
group name (not explicitly specified since group is an input parameter),
local rank, remote rank, channel name, and condition name.
The user specified name is the same for both ends of the channel.
b. Move all text after the create operation into channel overview chapter.
c. Add examples for the use of event names in the channel overview chapter.
d. Drop “MPIRT_CHANNEL_TRANSFER” for the name of a channel condition.
e. Percolate the changes for event object chapter (dropping handlers)
to this chapter.
The Collective channel chapter was read and voted in with official vote of 7/0/0 with
the following amendments:
a. Percolate all changes for Point-to-Point channels and channel overview.
b. Combined three tables of reduce operation into one and add C++ datatypes.
c. Synchronize with Commitable chapter.
d. Add bufiter with zero length buffers for barrier operation.
e. Update figures and their descriptions.
The commitable object chapter was read. While the chapter was not officially
voted on the following vote was taken after long discussion
and debate. There is a single commit/uncommit operation instead of
a three phase current proposal. Straw vote 8/7/0.
The old admit/transit proposal with discussed variation of separate
or combined Transit and Resource objects will be reconsidered for MPIRT-1.1.
Update figure for Init/Term chapter to reflect this decision.
The completable chapter was read and voted on. The official vote is 7/0/0
with the following amendments to the chapter:
a. Change Completable to Channel.
b. Merge this chapter with Data Transfer chapter and rename it
operations on the channels.
c. Change input parameter to allow operations on a channel and on a
container of channels.
d. Change discussion to a regular text.
e. Merge discussion after test operations with the text for the figure
for state transition diagram for a channel.
f. Synchronize the use of the status for the test with the state in
state transition diagram. Update channel state transition diagram
and define events as transitions between the states. Status represents
a state.
The data transfer chapter was read and officially voted in 7/0/0 with the following
amendments:
a. Add QoS error to the condition list, update the State transition diagram
figure.
b. Synchronize with channel chapters and event chapter (no more handlers)
c. Merge with completable chapter (before examples).
d. Convert discussion into advice to users.
e. Add C++ bindings.
f. Add pointers to QoS chapters.
g. Split examples into two subsection on for time-driven model and
one for event-priority driven one.
h. Update all examples and all example of the next chapter.
Several chapter and sections of the Real-Time Programming models and QoS.
– The names of all the communication paradigms chapter should be uniform.
– The section on Time Specification of the time-driven paradigm was
officially votes in with the vote of 9/0/0. The only amendment was
the change of MPIRT_DATATYPE_TIME_SPEC to MPIRT_TYPE_TIME_SPEC to
reflect that it is not a datatype to be transferred in a message.
– The Event-Driven Chapter was voted in by official vote of 7/0/2 with the
following amendments:
a. Move MPI references to footnote,
b. Change Comm to Group,
c. Change the name of the event as defined for the channels,
d. Add the example event names
– The Priority-Driven chapter was officially voted in with the vote of 9/0/0.
– Part of the previous vote was also to merge chapter 17 through 20 into
one chapter. This will leave this part with two chapters: one describing
paradigms and one for specification.
– The QoS Specification chapter was officially voted in with the vote of
7/1/1 and with the following amendments:
a. Change MPIRT_DATATYPE_TIME_SPEC to MPIRT_TYPE_TIME_SPEC,
b. Change Receptors to Event name,
c. Work out a proposal on adding time-spec for events (allow deadline
instead of timeout).

The meeting concluded at 12:15pm on May 21.

DRAFT Minutes of the MPI/RT Forum Meeting of April 8-10, 2022

Name/Organization e-mail
Manoj Apte, MSU manoj@erc.msstate.edu
Robert Babb, Denver U. babb@cs.du.edu
Dennis Cottel, SPAWAR dennis@spawar.navy.mil
Zhenquin Cui, MSU cuizq@cs.msstate.edu
Yogi Dandass, MSU yogi@erc.msstate.edu
Nathan Doss, Lockheed-Martin/GES ndoss@motown.lmco.com
Marc Fidler, Sanders marc.d.fidler@lmco.com
Robert Ginn, NAWC, AD rob@sun701.nawcad.navy.mil
Mike Grieco, ASEC/NSA mjgrieco@jswg.org
Shane Hebert, MSU shane@erc.msstate.edu
Randy Judd, SSC-SD, TASP judd@spawar.navy.mil
Arkady Kanevsky, MITRE arkady@mitre.org [chair]
Tony Lee, NAWCAD leeat%am5@mr.nawcad.navy.mil
Steve Paavola, Sky paavola@sky.com
Anna Rounbehler, Raytheon Anna_C_Rounbehler@res.ray.com
Alan Skillicorn, MITRE skilliad@cisf.af.mil

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, Sky, MITRE,
LM GES, Raytheon, NSA, Sanders, NAWC

Official vote summary for meeting:
Formal:

Initialization/Termination - Chapter [23], first reading, passed 9/0/0
Containers - Object section 4.6, 1st reading, passed 8/0/1
Generic Operations on MPI/RT objects - Object section 4.4, 1st reading,
passed 7/0/0
All parameters passed by reference and can not be modified/free after
commit if they are still in use by some committed object
passed 9/0/1

Straw [selected]:
MPIRT_OBJECT_TYPE - the name of the enumerator (replaces all
MPIRT_OBJECT_CLASS_NAME, MPIRT_OBJECT_CLASS_TYPE and other variations
of it). passed 13/0/1

The meeting came to order at 8:30am on April 8.

Tony is sick and not able to attend the meeting.
The meeting opened with a discussion of the newest draft, which was distributed
at the meeting, and put on the reflector as of April 7. A walk through
of the document was offered, with highlights and lowlights offered.
It was noted that several new chapters have appeared, several
have been moved, while others rewritten and “cleaned up” for reading.
The biggest changes are:
glossary has most of the terms now,
profiles and resource constrained chapters are moved from part I to
part VI,
MPI/RT object chapter is updated with new container section,
attribute caching section moved from communicator chapter, and
new UML object hierarchy diagram (appeared at the end of the document
due to the problems with postscript),
datatype chapter was copied from MPI-1,
light-weight handler section (created during the meeting),
committable objects (to be renamed Resources) outline created,
completable objects chapter written,
channel chapters updated,
transfer examples updated,
all chapters of QOS rewritten to reflect February meeting votes,
initialization and termination chapter rewritten,
instrumentation (was distributed during the meeting),
resource constrained MPI-1.0.
Shane presented Inertia. Inertia 0.5 is ready and available.
MSU also supplied several MPIRT example that run on it.
See http://www.erc.msstate.edu/labs/hpcl/inertia.
Yogi will create a link from MPIRT web site to it, and
will add examples into MPI/RT document.
Reading of the “Objects” chapter #4 (old chapter 5) was undertaken.
It was read every day with an updated version provided by Arkady, Shane
and Yogi dayly.
– The new UML diagram is separately presented at the end of the
document.
After a long discussion on classes and names following resolutions
were made on the first three sections of the chapter:
a. Drop “object” from user_object and system_object on
the diagram.
b. Remove User and System objects ancestors from the diagram.
(this was done on the second day which explains redundant
previous resolution.) straw vote passed 7/2/3.
b. Make the use of object class, object, object type and
object instance consistent through out the document.
c. Drop MPIRT prefix from diagram (it only appears at the root).
d. Remove the clock as a singleton object from the diagram
completely. This gets rid of the only singleton object and
document no longer need to describe singleton objects.
(it may come back if we decide to have SYSTEM object and
update function names accordingly (like MPIRT_SYSTEM_INIT).
It was discussed at the last day of meeting but no decision
was made.)
e. Define a virtual DUP at an ancestor level (old MPIRT_OBJECT).
C does not need to have a separate dup on each object level,
while C++ has virtual inherited dup.

– The generic operations on all objects were discussed in part
as a container discussion (section 4.4).
Following resolutions were made:
a. MPIRT_OBJECT_TYPE - the name of the enumerator (replaces all
MPIRT_OBJECT_CLASS_NAME, MPIRT_OBJECT_CLASS_TYPE
and other variations of it). straw vote passed 13/0/1.
b. Add operation MPIRT_OBJECT_GET_OBJECT_TYPE that is enhereted
by all objects (done during the meeting. included in the
official vote).
c. Add definitions of IN and INOUT parameters to the glossary.
d. All objects that are used as parameters are passed and
stored as references. The corresponding get_parameter call
returns the stored reference. The underlying object identified
by the reference can be modified (which effects all objects
that reference it) unless it is committed. (itself or via
some committed object that references it). As soon as
an object is committed its parameter objects are committed.
Official vote passed 9/0/1.
e. The user does not need to “free” the reference he/she obtained
from the get operation (see previous item). However, it is
user responsibility to free the objects that are no longer
needed and are not in use. A single free destroys
uncommitted/noncommitted object. Free is shallow and does
not work on committed objects. Straw vote passed 9/1/5.

        Formal first vote on section 4.4 passed 7/0/0.

– The first reading of container section (4.6) took place (actually
there were three readings of it, one per day). The following
changes were agreed upon, made during the meeting and voted in:
a. Remove “size” from Container_set.
b. Add “name” parameter to container object (every MPIRT object
has a name (of type MPIRT_STRING_NAME), and a type
(of type MPIRT_OBJECT_TYPE).
c. Add “count” attribute for a container (both set and vector
types). Straw vote passed 13/0/2.
d. Containers do not support built-in datatypes.
e. Define homogeneous. Remove wording that containers are
homogeneous. If a container contains objects of different
types one can walk up the object hierarchy to the topmost
level in the worst case to get every object “homogenized”
to MPIRT_OBJECT in the worst case.
f. Remove “generic” container object and split containers into
sets and vectors.
g. Return error if element to be removed is not in the set, or
currentelement for retrieve_next is not in the set.
h. Add function to query of a reference for an
element is in the set.
i. Return error if duplicate reference is trying to be
inserted into a set. Straw vote passed 10/1/1.

        Formal first vote for the section 4.6 passed 8/0/1.

The event chapter #8 was read for the first time. This is a new chapter
that presents events: triggers and receptors as agreed at the last meeting.
The discussions took a long time slipping into QOS topics and into
communicators discussions. No formal vote was taken due to the flex of
communicator/group chapter that is needed to define scope of events.
Never the less, the following changes were agreed upon:
```
      a. Triggers and receptors need generic object names (as all
    
```

MPIRT objects (local), and they need a globally unique name
for matching two sides of an event (trigger/receptor).
The event name is used for the second one.
b. The scope of the event delivery need to be reconsidered.
Existing groups and communicators definitions are not sufficient
for it.
c. Two events with the same QOS should be delivered in
the pairwise order. (If they are generated in some
rank and are delivered to the receptors of identical rank.)
This has no effect on this chapter but will be considered
at the next meeting when we will read QOS for events.
d. Add functionality to enable/disable receptors.
Disable receptors drop incoming corresponding events.
No queuing events on disable receptors.
Default value is enable.
This creates one more parameter for receptors and
one more function need to be introduced to get this
parameter (enable/disable status).
It is an error to enable abled receptor, and it is an error
to disable disabled receptor.
Straw vote passed 13/0/2.
e. Allow handlers for the receptor condition: receptor’s event
arrived. Straw vote passed 13/0/2.

The handler chapter #7 was read for the first time. The major changes
since last time is introduction of light-weight handlers and use
containers in parameter definitions. No formal vote was taken due to
the fact that people wanted more time to absorb changes and see all the
places where handlers are used. The following changes were agreed upon:
```
     a. System handlers need object name.
         b. Standard needs to define default behavior for conditions
    
```

for which there are no handlers.
c. There was a long discussion on two types of handlers,
one which is recoverable (support callback) and which is not.
The call back version is needed to support bufiter overflow
and overflow condition where the same insert/remove buffer
operation need to be repeated after the successful return
of a handler. Shane and Steve will write a proposal for
the next time. Steve proposed at the meeting a skeleton
of the handler implementation; Yogi will add it to the
document.
d. Details of the light-weight handlers are needed (current
text just provides APi). Example of its use for instrumentation
is needed.
e. Rename Object_type and Object_cond parameters
into anchor_type and anchor_cond in order to avoid function
name conflict with MPIRT_OBJECT_GET_OBJECT_TYPE that returns
the type of the calling object vs. the object that handler
is attached to.

The initialization and termination chapter #23 was read and voted for
the first time. Here are the list of changes that were made during the
meeting and voted on at the last day:
a. Add arguments definitions to function declarations
(argc and argv). (done)
b. Include advice to implementors about timeout. (done)
c. Define the state after finalize and before init.
Which processes are still alive. This takes care of an
MPI problem when MPI_INIT create processes and
MPIRT_FINALIZE destroys them. (done).
d. Update the first figure to show process before init, after
finalize, and add commit/transit operations to show
synchronization points before any user data transfers and
after it. Delete the second figure.
e. Add definition of the application process.

Chapter was formally votes in (first vote) 9/0/0.
As part of the resource constrain chapter the following global decision
was made:
a. Split the document into two volumes: volume I -
Standard - parts I-V,
and volume II - parts VI - VII.
No vote was taken but decision was unanimous.
The discussion on Resource constraint MPI-1.0 took place.
No formal votes were taken but many straw vote to guide the authors.
```
     a. There will be only one resource constrained profile for MPI-1.0
    
```
b. Remove MPI_Errhandler functionality. Keep MPI_ERROR_CLASS.
Remove now unnecessary error codes.
c. Keep MPI_COMM_DUP and SPLIT. (29.2)
d. Remove groups, COMM_CREATE, Inter_communicators, attribute
caching, collective computations with extended datatypes,
user defined collective computations, buffered send mode,
sendrecvreplace, error strings, process topologies, derived
datatypes (29.6, 29.7, 29.8, 29.10, 29.11, 29.12, 29.13,
29.14 and 29.15).
There was a discussion on the instrumentation chapter. No formal or straw
votes took place. Some agreements were reached:
a. Put the metric for 25.2 into a table.
b. Remove operations on name. They are inherited from MPIRT Object.
c. Rename operations. Get and Set are used only for object
parameters. Use retrieve for other gets for this chapter.
d. Consider rewriting 25.5 (generic probe) and moving it to
MPIRT-1.1.
e. Update API to reflect OO terminology used through out the
document.
Thanks were offered by the committee to Al Skillicorn for hosting this
MPI/RT Forum meeting.

The meeting concluded at 12:45pm on April 10.

DRAFT Minutes of the MPI/RT Forum Meeting of February 24-26, 2022

Name/Organization e-mail
Darwin Ammala, MPI Softtech dammala@mpi-softtech.com
Manoj Apte, MSU manoj@erc.msstate.edu
Robert Babb, Denver U. babb@cs.du.edu
Dennis Cottel, SPAWAR dennis@spawar.navy.mil [local host]
Yogi Dandass, MSU yogi@erc.msstate.edu
Clair Guthrie, Navsea guthrie_clair@hq.navysea.navy.mil
Shane Hebert, MSU shane@erc.msstate.edu
Randy Janka, GTRI randy.janka@gtri.gatech.edu
Randy Judd, SPAWAR judd@spawar.navy.mil
Arkady Kanevsky, MITRE arkady@mitre.org [co-chair]
Tony Lee, NAWCAD leeat%am5@mr.nawcad.navy.mil
Steve Paavola, Sky paavola@sky.com
Tom Robey, Khoral Research trobey@khoral.com
Anna Rounbehler, Raytheon Anna_C_Rounbehler@res.ray.com
Brian Schott, USC/ISI bschott@isi.edu
Tim Singleton, Navy Singleton_Tim@hq.navsea.navy.mil
Anthony Skjellum, MSU tony@cs.msstate.edu [co-chair]
Rick Steinberger, LM GES rsteinbe@motown.lmco.com
Clay Taylor, Jr, MPI Softtech cdtaylor@mpi-softtech.com

Organizations with official vote for this meeting:
MSU, US Navy/TASP, Spawar Systems Center, GTRI, Sky, Denver U., MITRE,
MPI Softtech, LM GES, Khoral, Raytheon

Official vote summary for meeting:
Formal:

Clocks[21], 2nd reading, passed 7/0/0
Objects[6] modulo section 6.4, 1st reading, passed 8/0/0
Chapter[15] QoS, first reading, passed 9/0/1
Chapter[16] Time Driven, first reading, accept 16.1 and 16.2, passed 10/0/0
Chapter[17] Event Driven, read, but no vote taken; because of major changes
Passed vote to use trigger/event formalism, passed 10/0/3
Combine commit & event registration, passed 11/0/2
All late-binding issues will be postponed to MPI/RT 1.1, passed 7/0/1
Straw [selected]:
Steve Paavola’s proposal for instrumentation will be the basis for further refinement.
Operations on objects are to be generalized to common base class for simplicity of
the overall API.

The meeting came to order at 8:30am on February 24.

The meeting opened with a discussion of the newest draft, which was distributed
at the meeting, and put on the reflector as of February 24. A walk through
of the document was offered, with highlights and lowlights offered by both
co-chairs. It was noted that several new chapters have appeared, and several
have been “cleaned up” for reading.

The glossary was discussesd and augmented as part of this.
The customary discussion of the TASP schedule, and MPI/RT’s progress was opened,
with emphasis coming from Tim Singleton about the need for prompt conclusion of
version 1.0. Skjellum and Kanevsky reiterated the positive progress. The TASP
need for Inertia was also discussed informally, but the longer discussion was
deferred to the Wednesday afternoon slot, in which Hebert would discuss Inertia.
The 2nd reading of the “Clocks” chapter #21 was undertaken, with some minor issues
raised, and then it was passed by formal vote of 7/0/0 (motion carried unanimously).
1st Reading of the “Objects” chapter #6 was undertaken.
– Section 7.6 is to be put into 6.4
– Various editorial fixes were offered; co-chairs suggested that they will
do English editing later, and get help from an English-editorial person
at the near conclusion of the effort.
– Besides grammatical flaws, various paragraphs were promised to be rewritten
for clarity.
– It was agreed that Figure 6.2 would be redrawn in UML [one of Tony’s graduate
students is working on UML for MPI/RT, so Diane Wooley will be on the
hook for this. Randy Janka also agreed to try something.]
– Despite modifications, Paavola moved and Kanevsky seconded (Skjellum was
chairing at this point) the proposal to take 1st vote on Ch 6, minus section
6.4, which will be reviewed and voted separately. The vote was 8/0/0 to
accept the 1st reading modulo section 6.4 (unanimous).
The committee in general asked that the editors depart from the standard headings of the
report style, and introduce these readability features, to which the editors promised
best effort:
– Page numbers at center bottom,
– headings left facing to improve finding out what is where
The first review of the Communicators/Groups chapter was undertaken, which had already
been modified to remove attribute caching to the Objects chapter.

– It was noted in this context that the new 6.4 (not covered under topic 4 above),
would apply to all children of MPIRT_Object
– the callbacks would be made commensurable with MPI/RT handlers; it was noted
that the original 1.0/1.1 version of certain MPI callbacks are flawed or
problematic, and that this would be reviewed carefully.
Some discussion of VSIP, streams, and p. 85 of document were offered. This led to
no specific conclusion, other than that it would be revisited on March 3-4 at the
Boston VSIP meeting, where the issues/modifications to VSIP would be covered.
Points about content-addressibility of handler access was brought up by Paavola.
Paavola wants to be able to “look up” handlers according to objects they attach to,
and conditions. Paavola was asked to write this.
It was agreed that all references to MPI-1 and MPI-2 would be put in descriptive
footnotes, rather than in running text, so that it would not be necessary to read
either or both of the MPI standards prior to learning MPI/RT. The rationale for
the footnotes remaining would be to offer perspective to those who already know
MPI, or who wish to make comparisons.
Point-to-point and collective “Channels” chapter #12 was read.
The following agreements (straw votes) where reached: the old note that qos spec for
two (or more for collective case) endpoints of the channel must be the same. The only
exception is a special constant MPIRT_QOS_TRANSFER_ACCEPT_ANY that is allowed to match
any specification of the qos from another end(s) of the channel.

The discussion on “strings” and names took place. We aggreed to have a datatype
MPIRT_NAME to avoid space allocation for the strings and leave this issue to implementation.
The name will still be INOUT parameter. These decision affected several operations
in this and other chapters including earlier and later chapters: event-driven, and
handlers. The part of the object hierarchy will apparently look like:
```
		MPIRT_String
          \
          MPIRT_Name
    
```
Where the MPIRT_String datatype may be used more generically elsewhere.
The straw vote was taken (passed by acclamation) to combine API to
the highest object in the hierarchy to reduce the number of functions in the document
as well as the size of the document. The point-to-point channel and collective channel
will be combined into a higher level object (name “channel” will be used for that object
and “ptchannel”) to accomodate that.

The meeting adjourned at 5:30pm on 2/24/98
The meeting reopened at 8:00am on 2/25/98

Chapter 15, QoS, was read and voted on 9/0/1 (formal vote passed).
Only minor rewording was suggested for this chapter.
Chapter 16, Time Driven, was read and voted (with exception of section 16.3) in 10/0/0.
Only minor rewording was suggested for 16.1 and 16.2. A portion of 16.3 that deal with
definition of MPIRT_TIME_OBJECT datatype will be moved into datatype chapter. The
remaining portion will be completely reworded. The new name will be chosen for the
“datatype”.
Chapter 17, Event-Driven, was read.
After prolonged discussion the following decisions were voted on.

There will be two committable objects: triggers and events (the names will be changed:
one approach, Event will be general naming; what used to be events will be triggers;
what used to be triggers will be ‘receptors’… this will be explored further and
clarified).

Each of the these have their own qos specification that is part of their specification.
Events also assign names that are also used for trigger specification to match with
the events. Formal Vote 10/0/3 (passed) on the functions/objects/behavior.

The commit (for channel sets) will be “combined” with event registration.
Also the same operation will “commit” other “commitable” objects including
channels via channelset, events and triggers (combined object will be used for that),
and “persistent generalized completable objects”. Vote 11/0/2

The editors will present the new operations (in the new committable objects chapter)
in the next version which is due at the beginning of the second week of March.
During the event discussion, Chapter 19 was also discussed. The previously mentioned
votes impact this.
It was requested that the API not specify the subclasses of MPIRT_QOS_TRANSFER
when naming this type of argument.
S. Hebert discussed Inertia status, work, and issues. He agreed to distribute the
first version about 6 weeks after the end of the meeting, matching the 3/13/98
release of the standard draft.

The meeting adjourned at 4:30pm on 2/25/98
The meeting reopened at 8:00am on 2/26/98 [S. Hebert presiding 8-10am, Skjellum
presiding 10am-12n.]

The Instrumentation Chapter was discussed at length. Paavola’s proposal was
introduced formally to the group. A straw poll was taken 11/0/3 (passed) to
accept and pursue Paavola’s proposal, and to see it completed.
It was discussed that sets should be contained in homogeneous containers. The
editors will impose edits to introduce this concept for next time. Examples are
as follows: probe sets, channel sets, event sets, PGCO sets.

Skjellum mentioned that it has always been the goal to replace naked arrays and
counts with homogeneous containers, and that this would improve the object-based
nature of the API.

Paavola asked about heterogeneous container objects, which apparently amounted
to encapsulating all the objects of a single MPI/RT “commit” and did not seem
to have much purchase with the committee.
Discussion of the level of completeness and timeliness of late-binding issues
was brought up. Skjellum offered the suggestion that late-binding be postponed
to MPI/RT 1.1, and suggested a formal vote. This was moved and seconded from
the floor, and passed 7/0/1. This will have certain impacts in several chapters.

During the debate concerning this, Skjellum described the general issues associated
with dynamic support and reconfiguration planned for MPI/RT 1.1, including extensions
and quality-of-service associated with MPIRT_CHANNELS_TRANSIT (mode changing). This
met with general approval.
It was requested that there be
– conditions defined for each object
– probes that are predefined
– system event names (POSIX subset) [later, we noted this is 1.1 only]
It was discussed that IMPI chapter would discuss interoperability of MPI/RT’s, not
interoperability of MPI’s with MPI/RT’s across implementations, nor refer specifically
to the NIST effort called IMPI.
The parliamentary procedure of approving the minutes at the subsequent meeting was
requested as point of order. The chair accepted that this would be the practice,
beginning with the current minutes, to be approved at the next meeting. The chair
also pointed out that these minutes would be subjected to web scrutiny and revision
long prior to that meeting.
Thanks were offered by the committee to Dennis Cottel for hosting this MPI/RT Forum
meeting.

The meeting concluded at 12:15pm on February 26.

Minutes of MPI/RT Forum meeting of January 14-16 in Albuquerque

Attendees:
Ron Brightwell, Sandia, bright@cs.sandia.gov (local host)
Arkady Kanevsky, MITRE, arkady@mitre.org (co-chair)
Zhenqian Cui, MSU, cuizq@cs.msstate.edu
Steve Paavola, SKY Computers, paavola@sky.com
Billy Chin, Lockheed Martin, GES, bill.k.chin@motown.lmco.com
Anna Rounbehler, Raytheon, Anna_C_Rounbehler@res.raytheon.com
Randall Judd, SSC-SD (USN), judd@nosc.mil
Dennis Cottel, Spawar Systems Center, dennis@spawar.navy.mil
Tom Robey, Khoral Research, trobey@khoral.com
Michael Grieco, ASEC (NSA K414), mjgrieco@jswg.org
Anthony Skjellum, MSU, tony@cs.msstate.edu (co-chair)
Nathan Doss, Sanders, ndoss@sanders.com
Clay Taylor, MPI Software Tech, cdtaylor@mpi-softtech.com
Robert Babb, U. of Denver, babb@cs.du.edu
Shane Hebert, MSU, shane@erc.msstate.edu
Robert George, MSU, robert@erc.msstate.edu
Yogi Dandass MSU yogi@aurora.cs.msstate.edu
Randy Janka Georgia Tech randall.janka@gtri.gatech.edu
X. Fowler U. of New Mexico [TO BE CORRECTED ONLINE]

The next scheduled release of the document is January 30, two weeks after
the end of the January meeting.
Since the December meeting, two additional informal meetings were held…
One at MSU (R. Brightwell, A. Kanevsky, T. Skjellum, C. Taylor, Z. Cui).
Second meeting at HICSS’98 in Hawaii (Arkady and Tony).
Rules for voting: One vote per institution.
Voting institution must have attended at two out of last three meetings.
There are two readings per each chapter.
A reading and vote within proximity of each other is preferred.
The final vote for each chapter of the document at the last meeting.

We consider national laboratories to be separate entities as
compared to their management company’s other activities for purpose
of counting institutions.

Clay Taylor gave an update on Data Reorganization January 13 activities
-Presentation by A. Rounbehler on Design For Data Reorganization API was given
-Discussion of data reorganization attributes
-Discussion of object design for data reorg and relationship to VSIP and MPI/RT.
VIA website is www.viarch.org. Via has slots for QoS but no
formal Real Time Model yet on its channels. Interesting possibility.
Status of the Inertia Implementation
Latest changes to the document have impacted the delivery of inertia.
Some Questions were raised by Shane:
What happens if a commit is called. Each time it is called, what resources
are available to the commit?
The old Transit operation need to be brought back.
What happens when inter-communicators are introduced.
It was decided that there is no problem,
since relationships occur between communicator groups.
Changes to the document, discussions and readings:
The January 12 version was posted before the meeting and
new January 16 version that included many updated chapters was
posted during the meeting on the web.

-C++ bindings have been partially introduced into document. These
will continue to be added from release to release, and will be discussed
at future meetings from a design perspective.

“Discussion” items now include answers when they are resolved.
They will be turned into permanent material in the chapters or removed
depending upon their significance.

All examples are not up to date. Do not take examples in the text
as representative for now.

Preface - New chapter, TBD for next meeting
Glossary - asked for during the meeting and placeholder chapter was created
in January 16 version. Names of terms and proposed definitions welcome
from all.

Part I - Overview:
A. Introduction for Users - new chapter, TBD for next meeting
Portions of the document “Summary of MPI/RT Basic Communication”
posted on the exploder will be placed in Introduction for Users.
The use of benchmarks for MPIRT. That work is in progress in MITRE, MSU and
Georgia Tech.
B. Introduction for Implementors - new chapter, TBD for next meeting
C. Introduction - new figure for two level middleware was added,
a text to accompany it will be added for next meeting
Reference diagram 4.2 (of Jan 12 version or diagram 3.2 for Jan 16 version)
Need to make changes to mnemonics (L1, L2, L2+). Need
words and descriptions that clarify the levels of middleware.

Part II - Concepts: major revision

A. MPI/RT Objects chapter was moved from Part I into Part II, and
completely rewritten. The chapter was read twice.
AT the first reading January 14 the following agreements were reached:
chapter will be divided into three sections:
Overview of objects, system objects, user objects.
Hierarchy diagram needs more details: needs both system and user objects,
needs to clarify system object classes that have two or one phase
initialization, that allow wait and test operations, that allow data transfer
operations.
Requests were discussed. Erequests are not needed. The word request does not
convey useful information. It was agree to create two system object classes:
completable (which support test and wait operations), and transfer (that allows
data transfer operations). For now all objects in class transfer are also
in class completable.

The new diagram for MPI/RT object hierarchy was discussed again January 15.
The two-phased initialization objects will be presented as shaded boxes
rather then separate class. Group, Comm, Datatype are added to the system objects.
A new class was created as a child to system object class: memory descriptor.
It currently has only buffers but in MPI/RT-1.1 scatter/gather buffers
will be added.

Vote to replace old hierarchy diagram with the one presented
pending some analysis by Tony & Arkady. 14/0/2
Vote to change request to completable object [wait/test] 10/1/7
Vote to put buffer back to object 12/0/4
Write up a gather scatter descriptor for MPI/RT 1.1 has been put on hold

The January 16 version contains the new write-up of the chapter
that was read on January 16 (due to its importance to the rest of the
document). Many minor changes were proposed with clarification of the
new object diagram, definition of DUP, and relationship to datatypes being
the most significant.
The new version will appear at the next release.

B. Communicators, Groups and Context chapter - was added.
Currently empty but will contain chapter 5 of MPI-1 with MPIRT prefix,
and with certain other modifications relevant to MPIRT format.

C. Datatypes - chapter will be added to included portion of the basic
datatypes of MPI-1 from chapter 3 plus new datatypes that MPI/RT introduces.
January 16 version has a placeholder for it already.

D. System handlers chapter - major revision from December.
Handler function is fully defined, fanctor introduced. All old discussions
resolved. Quality of service was clarified.
The chapter was read on January 14.
Need to add a clarification on when and how can the handler function can
be changed. Need to add state diagram for handler object.

Chapter revised add read again on January 16.
The failure handlers are gone. They are just handlers that are invoked
on error conditions. Had discussion how to use handlers for monitors.
There was a discussion on light-weight handlers for completion of
data transfer for transfer objects. The name “system handler” will be
changed to “handler”.
Need clarification on multiple handlers (may be from different libraries)
on the same condition of the same object. The order of handler schedule
is determined by QoS.

Need a few more operations to easy a search
for multiple handlers on the same condition, with the same handler function
and so on. Steve Paavola agreed to do that.

Introduce the object name for non channel objects. This will allow
implementors simple rules for matching objects.

E. Buffer management chapters - minor revision:
New diagram for buffer management and old discussions resolved.
No reading on this chapter was done only some discussion.
The new subsection for data descriptors with buffer objects will
be added for the next document.

F. Channels chapter - No major revisions:
No discussions or reading were done on this chapter.

G. Collective operation chapter has been completely rewritten:
Broadcast, Gather, Scatter, Reduce and Barrier, have been included.
Some of the operations needs two set of bufiters one for sending
and one for receiving.

This chapter was discussion on January 15.
Barrier needs to be converted into MPI/RT format with the collective channel
with QoS. Each of the collective operations needs its own set of
get/set, dup, free operations, not just create.

Need full details on how to implement a collective operation as a collection
of point to point channels. This potentially include the creation of
new bufiters, bufiter of different sizes over the same memory and sharing
bufiters between point-to-point channels that are created by implementors.
This is basically an advice to the implementors how collective operations
can be layered on top of point-to-point operations.

The following were also noted during a discussion;
The differences in the number of buffer iterators for the different collective
operations (broadcast, gather, reduce).
Some parameters make sense only for the root (as in MPI-1).
-Discussion on what additional attributes are needed in buffer iterators
to support collective communication. The conclusion is that none are required.
Handlers were discussed with respect to supporting collective communications.

H. Data Transfer chapter - is completely rewritten.
Discussed on January 14 together with MPI/RT objects chapter.
Rewritten in January 16 version to be in synchrony with MPI/RT object chapter.
MPIRT_DO is gone; Wait and Test is on completable objects, all
data transfer operations (MPIRT_START, MPIRT_ACTIVATE, MPIRT-DEACTIVATE)
are on transfer objects. The transfer objects are all completable.

Part III - Models, had not been updated.
It is out of synch with. the rest of the document.
The changes for MPI/RT object chapter has major impact on it.
It will be updated for the next version of the document.

Part IV - Application management: major revisions:

A. MPI/RT Initialization and Termination has been written.
It was not included in the January 12 version, it is in January 16
version. It was not discussed.

B. Clock Chapter have not changed over the last year.
Chapter was read on January 15.
The following changes were requested:
MPI_WTIME to be changed to MPIRT_WTIME.
Change globally synchronized to master clock.
Define which clock (master clock or local).
Clarify the use of local, synchronized, global clocks when they are used
in light of new master clock and definition of synchronized clock which is
local.

Add to the text, MPI/RT is required to use a synchronized clock.
Advice to implementers should state recalculate not reset the drift parameter.
Define constant MPIRT_WTIME_UNDEFINED and use it for skew,
accuracy and access time and all other places where implementation can not
provide a bound. That takes care of the caveat and discussion at the beginning
of the chapter.
Add advice to implementers to clarify that the platform
or the MPIRT services can support the MPI/RT master clock.
The definition for MPIRT_TIME_OBJECT will be moved to datatype chapter.
Need a definition for Epoch. Recommend using implementation defined
MPIRT epoch with extra function to find the offset from
the POSIX definition of epoch.

The chapter was updated for January 16 version and read one more time on
January 16.
Many minor word-smithing exercises were done.
The new datatype MPIRT_DATATYPE_LONGLONG will be introduced in datatype
chapter, which new datatype MPIRT_DATATYPE_TIME is using.
This is needed to separate time from dimensionless drift which uses
the same size datatype.
All measurements are in nanoseconds.
MPIRT_WTIME, MPIRT_WTICK, MPIRT_DRIFT, MPIRT_SKEW, MPIRT_ACCURACY,
MPIRT_ACCESS_TIME, and MPIRT_EPOCH_GET_OFFSET functions are moved
to the places in the chapter where they are defined.

The chapter was VOTED on. 11/0/0 (institutional [formal] votes).

C Instrumentation chapter - has been updated.
It was not included in the January 12 version, it is in January 16
version. It was discussed on January 15.

It was agreed to support one API for instrumentation and monitoring.
This API would provide a forum approved selection of instruments
for MPI/RT and have a design that allowed expansion in the
future to add other instruments.
This additional functionality is deferred to MPI-RT-1.1.
To avoid unnecessary over head, a wait and test object for monitors
has not been included. The ephemeral (erequest) has been removed.
The monitor can
use the handlers instead of the ephemeral request.

Part V - Non-Real-Time Services - no changes.

Part VI - Interoperability with other standards: major changes: No discussions.

A. MPI: a small subsection on MPI_initialize
and finalize was written in January 16 version. Still incomplete.
Need a discussion in the document that describes the progress guarantees
for MPI with MPI/RT.

B. IMPI: empty - Thom McMahon and SHane Hebert will contribute text
for February time frame.

C. VSIP: preliminary version appears in January 16 version. Puri
Bangalore and Randy Janka will meet to discuss before next meeting.

D. Data reorganization: empty - Clay Taylor will contribute text
by February meeting.

E. Packetway: empty. Robert George will provide text by February meeting.

F. VIA: new chapter. Preliminary version appears in January 16 version;
Rossen DImitrov will offer additions after he gets feedback.

G. COE: preliminary version appears in January 16 version.

Part VIII: MPI/RT-1.1 - new part that include parts that where moved from
old document per agreement at December '97 meeting.

Meeting adjourned at 2:30pm Friday January 16.

MPI/RT Forum Meeting Minutes December 11-12 2021 San Diego, CA

Attendees:
Ron Brightwell, Sandia, bright@cs.sandia.gov
Arkady Kanevsky, MITRE, arkady@mitre.org (co-chair)
Zhenqian Cui, Mississippi State University, cuizq@cs.msstate.edu
Steve Paavola, Sky, paavola@sky.com
Chris Hayden, Sanders (a Lockheed Martin Company), cphayden@sanders.com
Darwin Ammala, MPI Software Technology, Inc, dammala@mpi-softtech.com
Billy Chin, Lockheed Martin, GES, bill.k.chin@lmco.com
Richard Steinberger, Lockheed Martin, GES, rstenbe@motown.lmco.com
Anna Rounbehler, Raytheon, Anna_C_Rounbehler@res.raytheon.com
Randall Judd, SSC-SD (USN), judd@nosc.mil
Tim Singleton, Navy Peo USW, Singleton_Tim@hq.navsea.navy.mil
Dennis Cottel, Spawar Systems Center, San Diego, dennis@spawar.navy.mil
– local host
Keith Bromley, Spawar Systems Center, San Diego, bromley@spawar.navy.mil
Tom Robey, Khoral Rsearch, trobey@khoral.com
Michael Grieco, ASEC (NSA K414), mjgrieco@jswg.org
John Williams TASC (NSA K414), jpwilliams@jswg.org
Anthony Skjellum, Mississippi State University, tony@cs.msstate.edu (co-chair)

December 11,2021
I. Overview

Review of issues left over from previous meeting:
Discussion of Activate/Deactivate
Email comments that have not been incorporated
Status on Inertia, the MPI/RT prototype - Most of code is untested.
Buffer Management
Buffer sets 90% written
Buffer Iterator 90% written
Buffer Iterator Accessor 80%written

Channels
Handlers 80% written
Channels 60% written
Channel set 50% written

Collective Operations 0%
Data Transfer 50%
Quality of Service 80%

Time driven
Event driven
Priority Driven

Need to have more questions answered by Reflector
Linux DARPA work will be the basis of Action prototype
Maruti (time based scheduler) will be available
Need VXworks to be modified - use third party add-on.
Comments on Inertia by T. Singleton
Money is limited to a Fiscal year
Vendors feel that it is langishing
No coordination with VSIP
Responsibility of MPI/RT to integrate to VSIP
Recommendations - lock down and baseline the spec.
New schedule for release of Inertia.
Concern expressed for no formal votes taken
Need to show solid progress
Need plan to solve the VSIP - MPI/RT compatibility problem
Problems with copies of special buffers
Export Paradigm proposed. Buffers are visible to the user.
User can perform computation on channel buffers.-VSIP.
VSIP memory is not visible to the user
Comments by Tony -
Issues to resolve
Implementation of Admission tests will need more work
Advice to Implementers need to be added to the document
Goal is to finish the standard by the end of June
Other related efforts
DARPA funded efforts to investigate OS and real-time schedulers
DARPA effort to implement Maruti
Windows C2E for real-time
Need improved scheduler for various OS to support MPI/RT -
third party product with real time extensions
DII-COE is certifying Lynx as standard OS.
Tony and Arkady are developing a 3-day Tutorial on MPI/RT. Several hundred slides.
Need a Web Site with information that describes what is MPI/RT.
Arkady & Tony will present a “schedule” for products to be developed.
Desire to go to ANSI standard for acceptance upon completion of the standard.
Political problems of bringing MPI-1.x functionality into ANSI.

II. Discussion of MPI/RT:

Request and Object Handles.
A. Object types: user and system objects
Object contains: Object description, handle, user invisible physical object,
and potential implementation defined attributes.

Common System objects operations: Create, Free, Dup, Get, Set.

Commit operation builds physical object from its desription.

B. Requests
Agree with Tom’s email comment about nomenclature should be requests.
Hierarchy suggestion STRAW VOTE - passed
REQ
MPIRT_REQUEST MPI_REQUEST
Channel Collective channel Channel sets

Should we have unique START, WAIT , TEST, PROBE in MPI/RT?
TEST and PROBE are costly.
CANCEL is rejected.
Mercury has requested a polling technique - poll on completion - TEST.
Need a specially defined handler for this functionality.
Proposal for functionality for data transfer operations
for MPIRT_REQUEST (START, DO, ACTIVATE, DEACTIVATE).
Straw Vote passed.
START is fire and forget about it. DO is fire and the ability to monitor.
There are two events on a channel that are of interest (atomic operation - data
transfer completed end to end) and (local completion
where the local request was satisfied).
How does the concept of TEST fit into the concept of performance monitoring? - The user
would like to take action based on monitored conditions.

Should MPI-1.0 calls be mixed with MPI/RT?
The chapter on collective operations is very outdated and needs work.

III Discussion of Collective Operations, Channels and Requests
On hold

IV Handlers - Section 7.1

History
Desire to have more than a single handler for channel objects.
Local and global completion of data transfer operations conditions.
Start of Data Transfer needs to be added.
Need to define MPIRT_handler_fn arguments
What are the arguments? (object , communicator, handler functor)
Handler functor(extra state, condition and other parameters from handler create).
Handlers can have handlers which have handles (cascading handlers).
Example with failure function. Discussion on two level hierarchy.
Failure_handlers do not allowed to have failure handlers.
Handlers can be used for other objects:
See page 8 for list of objects - there will be handlers for all except bufset.
STRAW VOTE Passed
Need clarification of Commit Rules
Need clarification of DUP rules
Proposal: To replace extra state by attribute caching - problem
with embedded architectures.
Refer to MPI-1.0 section on attribute caching.
Proposal: Single attrribute (ptr, length, copy callback, delete callback)
The implementation must treat a DUP channel as if it were created using Channel_Create.
Meaning of Uncommitted, Committed and Dup are the same as for all MPI Objects:
Straw Vote 10 yes, 3 abstain

V Other Suggestions
Assign tasks to various members of the forum to write examples

December 12, 2021

I .Review of Discussions -
COMMIT can be undone with a FREE.
DUP can be done only on a channel that is not COMMITTED.
Undecided as to whether committed channels can be DUPed.

II. Logistics
Proposed Schedule (the complete schedule will be posted on the web
and on the reflector)
First Readings Other
January,2022 Chapter 2 becomes chapter 5
Chapter 1,2,3,4,6 Chapter 5 becomes 16

February 2022
Chapters 7,9,10,15 Chapter 7 System Handlers will be moved
19-22, 16

April, 2022
8, 11,12,13, Part VI (23-28)
Tutorials (4 planned)

Actions
    Remove chapters 18
    Designate 14, 16, 17 for MPI/RT II
    Add 2 Chapters (1 and 2)
      Advice/Tutorial Introduction
      Advice/Implementors

Future Meetings - will be extended to three days
Notes
The status of a chapter will be in the first few lines of the chapter
Chapter 8 - collective operations (broadcast, scatter, gather, reduce, barrier)
Reshape in MPI/RT2
Chapter 16 Instrumentation deferred to MPI/RT 2.0. Do we want to do this?

III Straw Votes
Proposed Schedule - Unanimous
Move Chapter 2 to Part II 14/0/2
Move chapter 5 to Part IV 15/0/1
Split chapter 7 14/0/2
Chapter 8 -Reshape deferred
(bcast,scatter,gather,reduce,barrier) 14/0/2
Chapter 14 deferred - see 1.1 16/0/0
Chapter 16 Instrumentation deferred 3/6/6
Chapter 16 First Reading Feb. 12/1/3
Remove Chapter 17 16/0/0
Delete Chapter 18 14/0/2
Reorg VI and add layering MPI-1 chap 14/0/2

Need to address: How does instrumentation monitoring “fit”
into the PMPI monitoring concept?
Keith Bromley suggested we take advantage of Fault Tolerance and Handling
(Sanders, DARPA, JPL) efforts

IV Iterators
Recall that System Handler and Event Handlers differ.
Event handlers are more permissive.
System handlers are local while event handlers can be local and global.
Problems with System handlers: QOS, re-entrant problems
Intent of System Handlers - instrumentation, scheduling system resources
(this handler has x time units to run),
used with other objects (see page 32)
Proposal to change QOS for system handlers
Straw Vote -Proposal to make handlers functions re-entrant, thread safe,
full weight -advice to users. PASSED
Proposal to change arguments in QOS - agreed (no vote needed)
Page 34 - Shane pointed out that extra state has variable length arguments
which is a problem.
Will be re-written as part of the pervious vote.
What happens when there is an error on an object, if the handler function is called?
How do you recover from an error condition when QOS is not met?
How do you restart a channel?
How do you specify a recovery policy and access
the buf-iterators for control of the buffers?

V MPI/RT Initialization/Termination
A proposal for Chapter 13 Initialization is made by Tom (Khoral Research)
This seems out of the scope of MPI/RT.

A layered approached for MPI-1 and MPI-2 on top of MPI/RT discussed.
Need MPIRT_Init, MPIRT_Finalize, MPIRT_Finalized, and MPIRT_Initilialized.
Straw vote taken, passed unanimously.

Minutes of the MPI/RT Forum on November 6-7 in Boston

Attendees:

Randall Judd SPAWAR Systems Center judd@nosc.mil
Dennis Cottel SPAWAR Systems Center dennis@nosc.mil
Michael Grieco ASEC - NSA N41 mjgrieco@jswg.org
Thomas Robey Khoral Research trobey@khoros.unm.edu
Anna Rounbehler Raytheon Anna_C_Rounbehler@res.raytheon.com
Arkady Kanevsky MITRE (co-chair and host) arkady@mitre.org
Jerrell Watts Syracuse U. jwatts@scp.syr.edu
Shane Hebert MSU shane@ERC.MsState.EDU
Zhenqian Cui MSU cuizq@cs.msstate.edu
Manoj Apte MSU manoj@ERC.MsState.EDU
Tom Daggett NUWC tomg@mitre.org
Robert Cunningham MIT Lincoln Lab rkc@ll.mit.edu
Richard Games MITRE Richard_Games@iegate.mitre.org
Chris Hayden Sanders, Lockheed Martin Co. CPHAYDEN@mailgw.sanders.lockheed.com
Billy K. Chin Lockheed Martin, GES billy.k.chin@lmco.com
Richard Steinberger Lockheed Martin, GES rsteinbe@motown.lmco.com
Leonard Monk CSPI lmonk@cspi.com
Clay Taylor MPI Software Technology cdtaylor@mpi-softtech.com
Robert Babb U. of Denver babb@cs.du.edu

The following topics were discussed:

Dennis gave a short briefing on Data Reorganization meeting that
preceded MPI/RT Forum.
(see http://www.data-re.org coming soon).
For the future, we will coordinate MPI/RT meetings
with Data Reorganization meetings.
Shane presented the current status of IneRtia.
The discussion started on the progress engines in MPI, MPI/RT and
IneRTia implementation. It gradually spread on the issues of
thread and signal safety, enabling and disabling interrupts
at a user specified time… At the end it was recommended to Shane
to make appropriate decision for a progress engine for IneRTia
and document it properly; it may require an extra operation,
which is unique to the IneRtia, that users call to progress
implementation.
There was a prolonged discussion on objects in MPI/RT and their
potential interaction with MPI “objects”.
The following straw votes were taken:
“A channel is a kind of Request and can be used everywhere Request is”
(accepted 11/0/3);
“Get rid of MPIRT_Channel_get_request call”
(accepted 10/0/3);
“Add distinction between objects and handles
(objects that have handles (system objects), and other objects - opaque
(user objects))”
(accepted 15/0/0).

The last vote amounts to creating object hierarchy.
There is a need for up front chapter to describe it,
a need to clean the existing bindings and an email on
reflector to propose the exact terminology. (This was assigned to
Tony and Arkady.)

There was a discussion on committed and uncommitted “system” objects.
It was agreed that errors and clarifications will be added for
appropriate operations.

It was agreed that object FREE operations return NULL handle.
There was a discussion on buffer iterators and their functionality.
The main topics were blocking vs. nonblocking behavior, overflow and
underflow behavior, and insert/remove operations.

The following straw vote was taken:
“No error for no buffer available for MPIRT_Start & MPIRT_Wait
for either channel end should be raised. The QoS error may be raised”
(Accepted 10/0/3).

There was a discussion to associate “error/blocking” policy on
buffer iterator or channel.
No conclusion was reached, the issue will be revisited in San Diego.

It was agreed to add MPIRT_BUFITER_COUNT_BUFFERS(bufiter, bufcount)
that returns the number of buffers in the buffer iterator.

Several minor changes, like positive vs. non-negative for various counts,
were agreed on.

It was agreed that buffer addresses are not unique and the same
buffer address can be used for multiple bufsets to support
data fusion. The appropriate text and examples will be added.
The data transfer operations were discussed.
The following straw votes passed:
“The document will contain three distinct functionalities:
Start and Wait (MPI-1 like),
MPIRT_DO (start without wait),
Activate and deactivate (user will not start or “do” at this end of
a channel, the other endpoint of the channel, time calendar, predefined
events and so on will start the channel data transfer)”
(Accepted 13/1/2);
“Allow multiple channels to be active when they share iterator”
(Accepted 12/0/3);
“The QoS specified on two ends of the channel can be different”
(Accepted 13/0/3).
There was a discussion on performance for MPI-1 type functionality vs.
degrading performance (tightness of achievable schedules) and predictability
for RT functionality.
There was a request to add index to the document and a section on
terminology and definitions.
It was agreed that we will switch to the three day format starting
from the Albuquerque meeting and until there is a need for it.
The Albuquerque meeting will start a day earlier on Jan 14.
There was a discussion on the Momentum (the first real-time supporting
implementation prototype). The main topic was what to include in it.
The following areas were discussed: preemptive scheduler (at time …),
preemptive handlers, clocks and timers, profiles and instrumentation,
and an ability to pass to implementation
allocated resources and schedules.

The meeting adjourned at 3pm. on Friday, November 7.

Minutes of the MPI/RT Forum on September 24-26 at Mississippi State University, Starkville, MS

Attendees:

Jerrell Watts Syracuse U.*
Ron Brightwell Sandia
Michael Grieco TASC*
Thomas Robey Khoral Research*
Dennis Cottel NRaD*
Arkady Kanevsky MITRE (co-chair)*
Yogi Dandass MSU
Robert George MSU
Shane Hebert MSU
Greg Henley MSU
Anthony Skjellum MSU (co-chair and host)*
Zhenqian Cui MSU
Peter Brennan MPI Software Technology*
Darwin Ammala MPI Software Technology
[* = institutions present that are able to vote in formal votes at
this meeting.]

The following topics were discussed:

The document has been reorganized as a standalone standard,
as has been discussed on several occasions. The outline was
reviewed, and further structural changes were made between
Wednesday and Friday, including the introduction of “parts”
to the document in order to group sets of chapters. Several
new placeholder chapters were introduced to help firm up the
scope of what the final document is to cover.

A number of changes discussed below have been effected during the
meeting, and an interim release of the document is anticipated early
next week.

The Inertia prototype was discussed (Shane Hebert presented).
The problems with buffers, channels, and templates were examined,
with recommendations to the committee for correction.
Templates were discussed, and it was voted to remove templates
in favor of new, but upward compatible technology from Jerrell
Watt’s recent write-up (distributed at the meeting). Shane had
also expressed desire prior to meeting to get rid of templates. These are
object descriptors (which are part of the opaque objects).

There will be a one-to-one correspondence of object descriptors and
their underlying real objects. Straw vote: 9/0/0. As a consequence,
channel and channel_set objects cannot be committed twice.

An object descriptor’s parameters cannot be changed once the object
is committed.

Discussions of “dup” (shallow vs. deep copy) ensued. Shallow copy
will be supported. [Straw vote on shallow vs. deep: shallow
accepted by 6/0/3.]

Discussion on adding a “deep dup” ensued. Failed by straw vote of 1/4/3.

Closure on the issue of buffers, iterators (replacing bufqueues),
and buffersets led to the clean model we have been seeking over
last few months.

Buffers will have ranks in buffersets, as well as labels.

Handlers were discussed. We voted to merge handler & error handler
classes. Straw vote 6/0/1. We decided that error handlers are of type
“system handler” see below, discussion item #14).
Prolonged discussion on buffers, buffersets, iterators, etc, was
undertaken. We discussed two iterators per channel, labeling of the
buffers (by integers), and indexing of the buffers by iterators.
The following straw votes were taken:
a. memory allowed to be used for multiple buffers: 8/0/3 in favor
(advice to users, advice to implementors will describe this)
b. we decided that two new statuses (stati?) or events will be
added: “when user asks for a buffer and the iterator is empty”
and “when buffer is returned to the iterator and the iterator
is full”
voted (unanimously): 11/0/0

This defines non-blocking behavior.
Channels were discussed, and new operations to activate or deactivate
channels or sets of channels were proposed. These are, in essence,
to start or stop schedules that were created by admission criteria.
[Supports mode changes.]

Straw vote was taken to remove old, non-template version of the
channel semantics. Carried: 8/0/0.

Quality of service was discussed. If the periods are defined for
the time-driven model, then relative start and deadline are defined
with respect to the start of the period, not the time at which the
operation was posted. Otherwise, start events or start have be an
absolute time or an event, and the deadline, if relative, is to be
with respect to the start. We allow deadlines to be longer than
the period, with appropriate advice added to the document, defining
issues concerning quality of implementation.
Jerrell/Tony will meet soon (September 27) to make further updates
to the document for the new style of collective channels that have
been agreed on, including exploitation of the iterators. This will
lead to support for the MPI-type collective operations in the face of
MPI/RT channels. This discussion will include review of the basic
API discussed under item 6.
The recent proposal of Anna Rounbehler/John Lebak was discussed,
and people promise to give feedback. The description of the buffers
and how the data is distributed across the communicator were missing.
The indices currently present are seen to be insufficient. The mapping
to the actual memory where data resides locally is unclear.

In addition to the needed changes, the updates reflecting our new gestalt
with collective operations will have to be incorporated.

Relationship to other standards (COEs, TASP, VSIP, IMPI, DARPA Data
reorganization) were covered.
We discussed the issue of MPI_, MPIRT_, and MPIFT_ headers, for
the base, RT, and fault-handling technology. We agreed to stay with
the various headers, as these make the most sense from the point of
view of portability.
We discussed a single handle for a channel_set, and decided to have
this. The main feature is that the request that represents the channel
set shall be analogous to a channel request. Semantically identical
to collective channel.
We agreed on the lower-level event paradigm is no longer in the
real-time paradigm, but fits into the “Concept” part of the new document.
There are two types of handlers: system handlers, and event handlers.
Profiles are behind schedule, and really will be in place for
November meeting.
Logistical issues:
next meeting at MITRE, November 6-7; coordination with other
meetings underway

following meeting at San Diego (NRaD or hotel); coordination
with VSIP has been done, other coordination under way
The www.mpirt.org web page is now active. The associated majordomo
is setup, and a copy of existing Argonne list members will be added,
informed about the new list, and given the option to delete themselves.

Pointers from the previous web site, www.cs.msstate.edu/mpirt, point
at the new web site.
It was announced that Yogi Dandass will be responsible for document and
web page updates (yogi@aurora.cs.msstate.edu, 601-325-4601), and will
assist the chairs.

The meeting adjourned at 2:50pm on Friday, September 26.

Faithfully submitted,
Tony Skjellum & Arkady Kanevsky

Minutes of the MPI/RT Forum on July 24-25 in Boston

Attendees:

Joe Germann SKY Computers
David Schwartz Hughes
Gerard Vichniac Mercury Computers
Tim Singleton NAVY
Jon Hiller Science & Technology Associates
John Williams TASC/K41 (NSA)
Donald Shakley MITRE
Shane Hebert MPI Software Technology Inc.
Randall Judd NRaD
Thomas Robey Khoral Research
Anna Rounbehler Raytheon
Arkady Kanevsky MITRE (co-chair and host)
Anthony Skjellum MSU (co-chair)
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Jim Ionata NUWC
Robert Cunningham Lincoln Labs
Bob Ford MIT Lincoln Labs
John Ramsdell MITRE
Paul Husted MITRE
Chris Hayden Sanders
Nathan Doss Sanders
Mike Butler MITRE
Richard Games MITRE
Stephen Rhodes Advanced System Architectures
Billy Chin Lockheed Martin, GES

The following topics were discussed:

Tony gave an overview of the current status of MPI/RT Forum.
The highlight of that was a schedule with deadlines for various sections
of the document. As usual the talk will appear on our web
(http://www.cs.msstate.edu/mpirt) soon.

The schedule for chapter completion will be put on the web site
prior to the next meeting as well.
For the future, we will coordinate MPI/RT development and meetings
with VSIP and TASP OS development and meetings.
Templates were discussed in details.

The view of the templates from the OO class point of view will be added.
The templates for individual object then become derived subclass,
which share the same set of operations.
We voted to delete “modify” and “create empty templates” through out
the document.
We voted to add “free” operations that allow to delete a template
instance through out the document.
We discussed operations that allow application to set up the default
values for template classes and derived classes. The pro and cons
of this preproposal were discussed, with the majority opinion that
users already have this functionality using “dup” and “set” operations.
The full proposal on that can still be entertain at the next meeting.

We discussed the possibility of reducing the number of “set” and
“get” operations currently in the document by having a single “set”
and “get” operation with one extra input parameter which specifies
which template parameter to modify. At the end the group agreed to
keep the existing API for clarity of use.

We agreed to change the names of the template operations to keep
“template” and template object name together and move the action
to the end of the operation name (e.g., MPIRT_TEMPLATE_BUFPOOL_CREATE).

The name bufpool will be reconsidered with potential change to
BUFFERSET. This may trigger further changes in the operation names
trying to achieve consistency in the naming convention.
The binding issue was partially discussed concentrating on delaying
FORTRAN binding, and potentially adding JAVA binding in the future.
The clock attributes were discussed again. The group voted to
make current attributes read-only. The scope of the synchronized clocks
was discussed, old idea of having attributes per communicator was
reintroduced, as well as when the clocks should actually be
synchronized. The proposal on this topic will be considered at the
next meeting, but may not be part of the December '97 MPI/RT complete
draft.
The profiles were discussed in great length.
We voted to add subset of MPI-2.0 instead of full MPI-2.0 functionality.

At the end the following 4 profiles were voted in:
- Resource constrained MPI-1.2 and MPI/RT,
- MPI-1.2 and MPI/RT,
- Resource constrained MPI-1,2, subset of MPI-2.0, and MPI/RT, and
- MPI-1,2, subset of MPI-2.0, and MPI/RT.
The resource constrained MPI-1.2 in itself is not a real-time profile
of MPI.

The subset of MPI-2.0 to be supported will be determined in future
discussion. It is anticipated to emphasize dynamic process management.
- There were multiple requests for including a list of functions as part
  of the definition of
- resource constrained MPI-1.2 systems.
- MPI-2.0 subset.
- It was suggested and agreed upon by consenus to give each profile a
  specific name.
Discussion of buffer pools (now officially called BufferSets) was
lively. It was determined after significant discussion that clarification
of the behavior under store-and-forward is needed. It was decided
that there may be some issues regarding “implicit” mode. It was
argued that the BufferQueue objects need to have a well-defined
initial state, and that clear functionality for both the get and
put side must appear in the standard.
An overview of PacketWay was given by Tony in regards to providing
interoperability services for several MPI or MPI/RT implementations.
[It was again commented that interoperability is part of the
MPI/RT effort.]
It was discussed to add the ability for handlers on send and receive
side to inherit the priority of the channel they are transmitting on.
[This makes sense definitely for receive side, send side TBD.]

Issues concerning the QoS of handlers
* time to begin execution
and
* duration of execution OR priority of execution
was given, commenting that these two issues are orthogonal. The
ability to guarantee a time bound for the commencement of execution
was debated.
Discussion of restructuring of the document into chapters occurred;
it is agreed that for September, the fully restructured document
would be offered. It would be offered actually prior to next meeting,
by the deadline of August 27. This will give significant time for
pre-meeting discussion on the reflector.
The need for an extra day of meeting (2.5 days) starting with September
MSU meeting was discussed and accepted. The meeting at MSU will be
so scheduled.
It is noted that consideration of VSIP’s buffering approaches
should be undertaken (cf, www.vsip.org)

VSIP attendee Dave Schwartz noted that future processors may not
offer single address spaces, and that VSIP explicitly did not
make such special memory available to users of VSIP (e.g., SRAM pages).
We intend to put online resources on the MPI/RT Web page, related
to: TASP, VSIP activities. [Tony to do this].
STRAW VOTES
Vote to remove template modify operation 10/0/1 carries
Vote to remove create_empty operation 10/1/0 carries
Request for class-based initialization of templates 2/6/10 fails
Add template free operation 14/0/4 carries
Make Clock parameters (section 8.5.4) read-only 11/0/6 carries
Define 4 profiles (see above) 15/1/2 carries
(Resource Constrained MPI 1.2 is not MPI/RT!)

The meeting adjourned at 12 noon on Friday, July 25.

Minutes of the MPI/RT Forum on June 5-6 in Boston

Attendees:

Dennis Cottel NRaD
Thomas Robey Khoral Research
Anna Rounbehler Raytheon
Arkady Kanevsky MITRE (co-chair and host)
Anthony Skjellum MSU (co-chair)
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Jim Ionata NUWC
James Lebak MIT Lincoln Labs
Rob Steele MIT Lincoln Labs
Daya Atapattu Scientific Computing Associates
John Ramsdell MITRE

The following topics were discussed:

MSU had established the web page for MPI/RT.
See http://www.cs.msstate.edu/mpirt.

The JOD version of our chapter is there in .ps and .pdf formats.
All past and future drafts will be posted there in .ps and .pdf formats.
Future meetings schedule is there as well as other MPI/RT related info.
We will investigate the possibility of rescheduling MPI/RT meeting
to be back-to-back with VSIP and TASP OS meetings. More on that
via email, and on the web site under “Meetings”.
The channel proposal was discussed.
We agreed on a layered template approach with user incrementally building
channel and channel set information with a single commit operation
for a channel set over a communicator. No actual objects like channel
requests, channel set handles and others do not exist until the commit
operation is performed. Only templates and handles to them exist
prior to that. This allows a very clean layer library design as well as
object hierarchy, while actually enhancing the optimizations possible
by MPI_Channels_Init et al.
The buffer management was revisited again.
Following the above layered approach the templates were created for
buffer pools and queues that are passed into channel templates.

According to the revisions made, users are allowed to create their
own buffers and pass addresses into a template. Users can also
request for a system to create buffers that will not be created
till the channel set commit is returned. Analogously, for queues,
users can provide persistent generalized requests (user defined
functions) for queue operation, or specify one of the MPI/RT
provided ones. The queue handles will not be return till the
channel set commit is returned.

The buffer access operations will be modified, and a new “true size”
operations will be introduced.
As part of the generalized request discussion related to the queue
management, “start” events will be reintroduced for lower and application
level event-driven paradigms.
The profiles and the relationships between MPI/RT, JOD, MPI-1.2 were
discussed and the new figure presented. This discussion will be
added to the next draft and the figure will be cleaned up.
We agreed that there is only one class of priorities, hence priority
class bullet for QoS for priorities will be removed. We voted in that
preemption quantum can be specified in time units or message length
units.

We also voted to merge priority paradigm with event-driven one. This
allows to specify priority as QoS for event handlers and data transfers.
We agreed to investigate full-weight and light-weight handlers.
The light-weight handlers will not use call back mechanism but will use
polling.
For application level event-driven paradigm we will investigate
receiver based model with “labeled/tagged” messages.
Collective operations were discussed. We agreed to pursue channel based
collective operations with QoS based on collective channel idea.
The transition of MPI-1 and MPI-2 operations with timeouts will be put
on the back burner in favor of real-time functionality and reshape
operation(s) which will appear in the document for the next meeting.
The MPI/RT draft still is not transitioned into stand-alone document.
This will be done prior to next meeting. Appropriate chapter divisions
for the first cut at the stand-alone document were discussed (again).

The meeting adjourned at 11am on Friday, June 6.

Minutes of the MPI/RT working group on April 25-26 in Chicago

Attendees:

Dennis Cottel NRaD
Thomas Robey Khoral Research
Anna Rounbehler SKY Computers
Arkady Kanevsky MITRE
Ron Brightwell Sandia National Labs
Robert Babb U. of Denver
Anthony Skjellum MSU
Harish Nag Intel,SSPD
Jerrell Watts Syracuse U.
Zhenqian Cui MSU
Peter Brennan MPI Software Technology Inc.
Tarek Abdelzaher U. of Michigan

The following topics were discussed:

Gordon High Performance Conference was announced.
See http://www.erc.msstate.edu/conferences/gordon97.
Tony will explore copyright transfer from U. of Tennessee
to MSU for future MPI/RT drafts and MPI documents.
A long discussion on the simulators of QoS for various platforms
including:
how to achieve it, how to use it, how to optimize it,
was conducted. This has a direct affect on implementation strategies.
Multiple levels of implementation activities were proposed:
API prototype with empty QoS guarantees, a generic QoS guarantee
for a neutral platform, and optimized QoS guarantee for real-time
platform.
The channel section was discussed.
A proposal was presented and discussed on creating objects to
reduce the number of parameters for channel management operations
and simplify their manipulations. The six page proposal was
written and distributed at the meeting and will be discussed at
the next meeting.

A split of collective channel operations into two stage process:
reserve and commit was proposed. A joint operation which
allows all channels (collective and point-to-point)
to get a complete picture on all needed resources collectively
over a communicator was also discussed.
The above mentioned object proposal supports this concept.
Several topics on Buffers were discussed. Buffers addresses as
IN/OUT parameters proposal was reintroduced. The vote was passed to pursue
the following model:
Nondestructive buffer send, with explicit wait and freeing the buffer.
All buffer operations are explicit (including all real-time data
transfer operations. Buffer iterators specified via strategy.
The other issues discussed were: who owns the buffers
application/implementation, buffers replenishment for buffer pools -
buffer passing between application and implementation, sharing buffer
pools between channels, buffer pool strategies and their specifications.
A discussion on mismatches of QoS for two endpoints of the
point-to-point channels, and multiple endpoints of collective
channels was conducted.
For the client-server paradigm only one end specifies QoS and the
other end accepts/rejects it. An analogous situation exists
for the root of a collective operation.
The current MPI/RT chapter will become a stand-alone document.
There was a discussion on the structure of the new document.
The new chapters will follow the logical breakdown of the
sections in the current document. The exact mapping of sections
to chapters will be discussed at the next meeting.
Additional chapters could include: introduction, dependencies on
MPI-1,2, collective operations, objects/handlers/request and other
underlying principles, threads.
Fault-tolerance will be renamed to fault handling.
Future of the MPI/RT was discussed.
We agreed to continue meetings on the 6 week basis.
Here is the tentative schedule of the future meetings:

Dates		Place
June 5-6		MITRE, Boston		Arkady Kanevsky,
August 7-8		MITRE, Boston		Arkady Kanevsky,
September 25-26	MSU			Anthony Skjellum,
November 6-7	MITRE, Boston			Arkady Kanevsky,
December 11-12	NRaD, San Diego		Dennis Cottel,
January 15-16	Sandia, Albuquerque	Ron Brightwell,
February 26-27	NRaD, San Diego		Dennis Cottel,
April 9-10		U. of Denver,		Robert Babb,
May 21-22		INTEL, Portland		Harish Nag,
June 29-30		MITRE, Boston	Arkady Kanevsky.

All meeting are 2 days: Full day first day and half day the second.
    
    The BOF meetings or tutorials will probably be held at
    SC'97  (High Performance Networking and Computing Conference) and 
    RTSS'97 (Real-Time Systems Symposium) conferences.
    
    We agreed to change our meeting name to MPI/RT Forum.
    
    A more rigorous voting and appeal rules will be introduced next time.
    
    The JOD will be published at the same time as the MPI-2 standard document
    as a stand-alone document that contains a chapter on MPI/RT.
    The current version will be presented there with the following changes:
    
    A. The first figure in the JOD MPI/RT chapter will be replaced by the
       current figure in the stand-alone chapter.
    
    B. Delete C++ binding (The group voted on it unanimously).
    
    C. Add disclaimer and the pointer to the ftp/html site of MPI/RT.
    
    D. Revert Buffer Management section to the previous version.
       Add the state diagram to it.
    
    E. Fix Appendices. Only the second appendix should be present that
       will be numbered A.
    
    F. Change Fault-Tolerance to Fault-Handling (voted in unanimously for
       JOD and all future versions).
    
    G. Put the two new figures from the buffer section that describe the
       sequence of application commands to the profile section.
    
    F. Add a figure to the profiles sections that illustrates relationships
       between, MPI-1, MPI-2, and MPI/RT (Venn Diagram).

WHERE TO LEARN MORE (AND GET ALL THE DETAILS – IF YOU WISH)

MPI/RT Standard Drafts

MPI/RT 1.1 Standard Draft (July 1, 2024 version)
Postscript Acrobat(PDF)
MPI/RT 1.1 Standard Draft (March 26, 2024 version)
Postscript Acrobat(PDF)
MPI/RT 1.1 Standard Draft (February 26, 2024 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 10, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 12, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (October 13, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (September 24, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (September 8, 2023 version)
Postscript
MPI/RT Standard Draft (June 7, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (March 5, 2023 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 1, 2023 version)
Postscript
MPI/RT Standard Draft (December 2, 2022 version)
Postscript
MPI/RT Standard Public Comment Draft (September 21, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (July 20, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (July 1, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (June 22, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Engineering Practices Draft (May 18, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (May 18, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (May 3, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (April 7, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 23, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (February 15, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 16, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (January 12, 2022 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 31, 2021 version)
Postscript Acrobat(PDF)
MPI/RT Standard Draft (December 11, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (November 25, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (November 3, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Standard Draft (October 3, 2021 version)
Postscript Gzipped Postscript Acrobat(PDF)
MPI/RT Draft (July 9, 2021 version)
Postscript Acrobat(PDF)
Realtime JOD Version (May 30, 2021 final version)
Postscript Acrobat(PDF)
Realtime JOD Version (May 8, 2021 first version)
Postscript Acrobat(PDF)

Errata

Items 1-8 were approved by straw vote on June 6, 2023.

At 98/34: the number of subbuffers to create should be specified as a “positive integer” rather than just “integer” since negative is obviously not right, and zero is not useful.
At 98/25 the Standard says that after calling MPI_CREATE_SUBBUFFERS, the new buffers are named with the name of the vector appended with the index of the buffer in the vector. The next sentence says:
“For example, the first created buffer in the V-container named bufcont has the name bufcont1.”
I believe this is a typo. The new name should be “bufcont0” since the indices start with 0.
I propose an errata entry to change the word “bufcont1” to “bufcont0” at 98/27.
It is necessary to be able to compare variables of type MPIRT_Dataspec. At 51/39 the 1.0 Standard says that MPIRT_DATASPEC_IS_EQUAL is not defined. This was an oversight/typo/mistake.
At 51/39, change “…set name, equality testing, and object decoration…” to “…set name, and object decoration…”.
Add to the table “Dataspec Operations” on page 63:
MPIRT_DATASPEC_IS_EQUAL MPIRT_Dataspec_is_equal MPIRT::Dataspec::Is_Equal
It was intended that the initial_buffer_list argument to MPIRT_BUFITER_CREATE could be given as MPIRT_CVECTOR_NULL if the user did not want to provide any initial buffers. However, this is not stated in the text of the Standard. The following errata entry clarifies this.
At 103/32, append the following new sentence to the paragraph:
The initial_buffer_list parameter may be given as MPIRT_CVECTOR_NULL if no initial buffers are desired.
In the table “Event Delivery Abstraction Operations” at the bottom of page 87, MPIRT_EVDEL_GET_NAME and MPIRT_EVDEL_SET_NAME should be added.
The following corrects and clarifies several issues in the specification for QoS for time-driven channels. Some of this could be considered part of the Error Proposal, but the issues seem too closely intertwined to separate cleanly.
The last sentence of the paragraph at 182/7 doesn’t compute. Replace the sentence with:
“The value of MPIRT_TIME_SPEC_IGNORE for the deadline indicates that a deadline is not requested.”
Delete the last sentence of the second bullet at 182/13. The correct restrictions will be described with other restrictions at the bottom of the page (see item #6 below).
Since the release time argument is required, delete the words “if one exists” at 182/16.
Change MPIRT_PERIOD_IGNORE to MPIRT_TIME_IGNORE at 182/22 and 183/5. Remove MPIRT_PERIOD_IGNORE from the tables on pages 205 and 206. This is left over from earlier drafts.
Add a new paragraph at 182/38:
If period is not MPIRT_TIME_IGNORE, it must be positive; otherwise the function will return MPIRT_ERR_INV_TIME.
6. Add a new sentence to the paragraph at 182/39:
In this case, the release_time must be non-negative and less than or equal to period; otherwise the function will return MPIRT_ERR_INV_TIMESPEC.
Add a new paragraph at 182/45:
If both release_time and deadline are of type MPIRT_ABSOLUTE, deadline must not fall before release_time. If release_time is of type MPIRT_RELATIVE, it must not be negative. If deadline is of type MPIRT_RELATIVE, it must not be negative. If any of the time specifications are incorrect or inconsistent, the function will return MPIRT_ERR_INV_TIMESPEC.
Add MPIRT_ERR_INV_TIME and MPIRT_ERR_INV_TIMESPEC to Annex A.
At 72/31, change “(MPIRT_TRIGGER)” to “(ref. to MPIRT_TRIGGER)”.
The following error codes are missing from the MPI/RT 1.0 Return Code Index on page 252 and the Return Codes Annex on page 244:
MPIRT_ERR_INV_QOS_CHANNEL
MPIRT_ERR_INV_QOS_CHANNEL_TIME
MPIRT_ERR_INV_QOS_CHANNEL_PREV
MPIRT_ERR_INV_QOS_CHANNEL_PRTIEV
MPIRT_ERR_INV_QOS_TRIGGER
MPIRT_ERR_INV_QOS_RECEPTOR
MPIRT_ERR_INV_QOS_HANDLER
Page 15 23/25: Modify statement to make it not as strong as it is because even a development library may not be able to detect all errors. Replace the sentence “All of the errors defined by the standard are required to be detected and reported, in order for a library to be called development” with “A development library should make best effort to detect and report all of the errors defined by the standard.”
Change the QoS parameter in all the collective channel operation specifications from “qos” to “channel_qos”.
Page 67: Add the following new paragraph and table at line 23:
For explicit triggers and receptors, the object name must be unique over the group and GROUP_WORLD. The scope for implicit triggers and receptors is GROUP_WORLD and the scope for all receptors and triggers must match.
Therefore:
Scope
implicit trigger GROUP_WORLD
implicit receptor GROUP_WORLD
implicit receptor for explicit trigger GROUP_WORLD
explicit receptor for explicit trigger Same as trigger
implicit trigger for explicit receptor GROUP_WORLD
explicit trigger for explicit receptor Same as receptor
Page 67 line 33: change “no” to “do”.
It was intended that the handlers argument to MPIRT_RECEPTOR_CREATE could be given as MPIRT_CONTAINER_NULL if the user did not want to provide any initial handlers. However, this is not stated in the text of the Standard. The following errata entry clarifies this. At 74/45, append the following new sentence to the paragraph: The handlers parameter may be given as either MPIRT_CONTAINER_NULL or an empty container if no handlers are desired.
In order to clarify which events are receoverable/retryable, delete the sentence at 75/12. Insert the following sentence as the second sentence of the paragraph at 75/7:
Recoverable conditions are those implicit conditions defined as recoverable in the standard.

The enumeration type MPIRT_SYNC_FLAG is missing from the entity table at the end of Chapter 4, page 90.
MPIRT_SYNC_FLAG | MPIRT_Sync_flag | MPIRT::Sync_Flag
Page 13, line 6 change “simila” to “similar”
Page 42 line 13 change “ne” to “new”
In figure 5.8 (pg 105), all three instances of the word “buffer” should be “bufiter”.
Page 148, there is a slight formatting problem in the table – the row for MPIRT_OP is incorrect.
Errata: General Comments 11/19/00
If we use the terms V-containers, vector containers, or VB-containers it is unclear exactly what we are referencing. The standard uses the term V-container, vector containers and VB-containers as follows:
Page 34/48 -“Vcontainers are derived from MPIRT_CVECTOR_BASE and are denoted VB_containers”.
Page 38/28-30 - VB_Container classes
Page 40/35 - “a group is a VB-container of process identifiers”.
Page 35/28-30 - vector containers are denoted MPIRT_CVECTOR_BASE.
Also see buffer iterator retrieve proposal.
34/46 Change the last word of the sentence from “V-container” to “VB-container”. Delete the confusing next sentence “V-containers are derived…”
35/25 Change "whereas vector containers " to “whereas VB-containers”.
MPIRT1.1 proposal authors should also check their proposals for consistent use of V-containers, VB-containers and vector containers terminology.

Each 1.1 proposal may need a few words in the Introduction . Here are some proposed new paragraphs and additions to existing paragraphs for the Introduction.
Page 9/3 The user may specify strided buffers, allowing access to non-contiguous blocks of memory.
Page 9/19 - At any time during the non real time or real time phase, the user may retrieve the number of buffers in the buffer iterator.
Page 10/11 - such as bipartite all to all, broadcast …
Page 10/12 - A collective channel utilizing strided buffers has a natural mapping for matrix transpose and 2D transformations.
Page 10/25 A program can use a receptor to wait for an event to occur.
Page 10/38 A receptor can be created with or without handlers.
The Introduction should be cross-checked after all MPIRT 1.1 proposals are finalized to be sure that the introduction is accurate with respect to the final versions of the 1.1 proposals (the introduction is high priority since it may be the only section that a majority of interested parties read).

Add the following definition to Glossary xx/25 Strided Buffers - A contiguous buffer partitioned into fixed length sub-buffers or blocks. The stride represents the number of elements between the blocks and supports non-contiguous buffer access.
Page 3/5 During MPIRT 1.0, it was proposed and approved in a previous errata that the following words be deleted since restrictions are implied that are not necessarily accurate: " MPIRT has complete control of resources and whose", .
Page 4/33-34 Correct the sentence to read " The tear down of these objects also occurs during a non-real-time phase"
In Chapter 1, we have lists of objects and/or classes in the text. Suggest that we avoid listing specific MPIRT functionality in this chapter wherever possible. The following changes are proposed to reduce the number of places that we have to keep updating as we add new proposals over time:
7/35 Change the sentence “some examples of committable classes are…” to “Committable classes are defined for buffers, buffer iterators, channels, handlers, and event delivery. Committable objects and their classes are described in Figure 2.1.”
8/40 Add to the sentence "The standard dataspecs supported by MPI/RT are integer, floating point, double precision, logical, character, byte and time. These dataspecs and corresponding language bindings are described in Table 3.1.
The Bufiter policies are not consistent in various parts of the standard. The correct list of policies is on page 111. The following changes are recommended:
9/12-13 Delete the sentences “Currently the standard …” and “Several more policies …”. Insert the sentence “The standard specifies LIFO, FIFO, sorted and unordered user-selectable policies; in addition to several other policies if the container is a vector container.”
9/13 Delete the sentence “these mapped and ordered policies …” and replace with the sentence “The standard specifies a number of user-selectable policies as listed in the table Buffer Iterator Policies in section 5.6.”
102/30 add the words MPIRT_BUFITER_UNORDERED
Search for all occurrences of MPIRT_BUFITER_ORDERED in the standard and remove from the document.
Need to clearly associate data transfers and channels with respect to zero, one and two-sided communication. The current sentences are too disjoint and need a lead in sentence. Suggest rewording with no impact to functionality as follows:
9/35 Insert before “Figure 9.1…”, the following sentence “MPI/RT supports zero-sided, one-sided, and two-sided communication for point-to-point and collective channels.”

13/11 Remove the words “In keeping with minimizing hidden dangers”
13/16-17 Replace “because libraries can be portably written by developers” with “because portable libraries can be written by developers”"
13/35 Replace " submitted by user" with " subjected".
13/37 Remove the sentence " and thereby the target hardware".
13/38-43 and 14/1-4: With no change to functionality, remove these 10 lines, starting with the sentence “Since the admission test depends…” and replace the paragraph as follows:
A successful admission test indicates that the target platform can provide the required resources within the desired performance parameters, based on the information supplied by the user. It is the responsibility of the user to provide the correct information. The user can obtain accurate information by experimentation. Using instrumentation, the user can start with a set of plausible criteria and iterate until the desired results are achieved. The criteria used to instrument may be a variety of heuristics or a single known quantity such as minimum frame rate. Whenever the platform cannot meet the user’s requirements, the admission tests will fail. When the admission test passes, the platform/ MPIRT performance tradeoff balance has been achieved.

63/11 Add a footnote for the function MPIRT_INT64_ASSIGN64 as follows: “This function is optional”.
23/41 Change “subsequent set” to “subsequent set operation”
24/39 Insert the phrase “by the user”, after the word “FREEd”.
94/40 Change “programmer” to “user”.

Proposals for MPI/RT 1.1

Intragroup all-to-all
Strided buffer
Errors
Error string
Container iterators
Sleep
Bufiter query
Receptor wait
Contianer operations
Object is committed
Variable length buffer
Convert int64 to two 32-bit ints
Bipartite all-to-all

Test Suite for MPI/RT

The Real-Time Message Passing Interface (MPI/RT) Test Suite is a freely available, independently developed package for evaluating the conformance of an MPI/RT implementation to the MPI/RT Standard. The Test Suite is funded by the US Navy through the Naval Sea Systems Command (NAVSEA) PMS 411.

Although the Test Suite is under development and only partially complete, the current version is available for download here. Documentation in the download package includes a Users Manual, a description of the design goals and assumptions, and an up-to-date test specification.

For additional information, contact Dennis Cottel, SPAWAR Systems Center San Diego, dennis@spawar.navy.mil.

Download the September 25, 2021 version (mpirt_ts_25sep2024.tar.Z)

Talks & Papers about the MPI/RT Standard

TALKS

January 2021 MPI/RT Tutorial at MITRE (Kanevsky, Rounbehler, Skjellum) Acrobat(PDF)
March 2021 MPI/RT Talk (Skjellum) Acrobat(PDF)
DARPA Quorum 2021 Talk (Kanevsky/Skjellum) Acrobat(PDF)
PDPTA’97 Talk (see paper below) Acrobat(PDF)
3rd IEEE Conference on Engineering Complex Computer Systems Tutorial (Kanevsky) Acrobat(PDF)
HPEC 2021 MPI/RT Talk (Skjellum, Kanevsky) Acrobat(PDF)

PAPERS

First MPI/RT Paper [Jan. 2021] (Kanevsky, Skjellum, Rounbehler) Acrobat(PDF)
PDPTA’97 Paper (Cui, Kanevsky, Li, Skjellum) Postscript Acrobat(PDF)
RTSS’97 Paper [Dec. 2021] (Kanevsky, Skjellum, Watts) Acrobat(PDF) Postscript
HICSS31 Paper [Jan. 2021] (Kanevsky, Skjellum, Rounbehler) Acrobat(PDF) Postscript
EE Times Article [September 27, 2023, Issue: 1080] (Christodoulou) HTML

Standards efforts related to, useful with, or incorporating MPI/RT

TASP Tactical Advanced Signal Processor Effort

VSIP Forum Vector Signal & Image Processing Forum

MPI Forum MPI/RT Has its roots in MPI-1 and MPI-2,
whose standards are contained within this site.
The MPI Forum no longer meets.

Data Reorganization Standard DARPA is sponsoring an effort to look at high performance
collective data reorganization API’s and protocols, a complement and supplement to MPI/RT.

IETF PacketWay Standard
Several groups are working together to specify a high performance system area network (SAN) protocol,
including security, for interoperability, low latency, and high bandwidth. This is a key technology for
providing routing to support inter-MPI/RT implementation communication.

VI Architecture (Intel/Compaq/Microsoft)
A large group of industry interests is producing a SAN datalink layer specification (MAC + LLC)
called the Virtual Interface Architecture (VIA). VIA supports channels, and scalability.
With articulation of QoS on these channels, VIA could be an implementation mechanism
for MPI/RT channels on a SAN network, with PacketWay supporting inter-SAN routing.

Example code for the MPI/RT Standard

WHO TO CONTACT

Contacts (Standard co-chairs): Tony Skjellum Arkady Kanevsky
Forum pages/standard document maintainer: Yogi Dandass

OTHER INFORMATION & ACKNOWLEDGEMENTS

For more information on the base standards:

MPI-1, and MPI-2, upon which MPI/RT builds, see both:
MPI-1/MPI-2 MSU Page MPI Forum Archival Page
Major funding for MPI/RT is from DARPA

MPI/RT Real-Time Message Passing Specification based on MPI (MPI is the Message-Passing Interface)

GETTING INVOLVED AND KEEPING UP

MPI/RT 1.1 Topics

MPI/RT Forum Meetings

MPI/RT 1.1 Forum Minutes

Minutes of the MPI/RT meeting Held on December 13-14, 2023 in Orlando, Florida

Minutes for the MPI/RT Fourm meeting at MITRE, Boston on 18 September, 2023

Minutes for the June 6-7, 2023 MPI/RT meeting hosted by Mercury Computer Systems, Inc. in Boston

Notes from the March 9-10 MPI/RT meeting in Atlanta.

MPI/RT Meeting, September 24, 2023, held at Mercury Computers

Minutes from the June 8-10 2023 MPI/RT Forum meeting that was held at LM-GES in Moorestown, NJ.

Minutes from the April 7-8 2023 MPI/RT Forum meeting that was held in Bedford, MA.

Minutes from the Feb. 3-5 2023 MPI/RT Forum meeting that was held in San Diego, CA.

MPI/RT Forum Minutes

Minutes for the Dec. 2-4 2022 MPI/RT Forum meeting held at MITRE in Bedford, MA.

TOPIC | Expected | Expected | and person(s) in charge | Discussion | Completion|

Minutes for the 22 July 2022 MPI/RT Forum meeting held in Washington DC.

Draft Minutes for the June 23-25 2022 MPI/RT forum meeting held at Mississippi State University

DRAFT Minutes of the MPI/RT Forum Meeting of May 19-21, 2022

DRAFT Minutes of the MPI/RT Forum Meeting of April 8-10, 2022

DRAFT Minutes of the MPI/RT Forum Meeting of February 24-26, 2022

Minutes of MPI/RT Forum meeting of January 14-16 in Albuquerque

MPI/RT Forum Meeting Minutes December 11-12 2021 San Diego, CA

Minutes of the MPI/RT Forum on November 6-7 in Boston

Minutes of the MPI/RT Forum on September 24-26 at Mississippi State University, Starkville, MS

Minutes of the MPI/RT Forum on July 24-25 in Boston

Minutes of the MPI/RT Forum on June 5-6 in Boston

Minutes of the MPI/RT working group on April 25-26 in Chicago

WHERE TO LEARN MORE (AND GET ALL THE DETAILS – IF YOU WISH)

MPI/RT Standard Drafts

Errata

Proposals for MPI/RT 1.1

Test Suite for MPI/RT

Talks & Papers about the MPI/RT Standard

TALKS

PAPERS

Standards efforts related to, useful with, or incorporating MPI/RT

Example code for the MPI/RT Standard

WHO TO CONTACT

OTHER INFORMATION & ACKNOWLEDGEMENTS

MPI/RT
Real-Time Message Passing Specification based on MPI
(MPI is the Message-Passing Interface)

TOPIC | Expected | Expected |
and person(s) in charge | Discussion | Completion|