Here's an updated proposal for variable buffer offsets and offsets. The one 
issue raised at the last meeting that I haven't addressed is how this applies 
to collective and gather/scatter channels. I propose that these enhancements 
only apply to point-to-point channels. Significant enhancements would be 
required to make them work with collective channels, and I feel those 
enhancements would be too much for the 1.1 schedule. We would have to add 
error returns to those functions if

The proposed changes to the MPI/RT V1.0 functionality are as follows:

BUFITER MODES

In order to support variable length transfers and variable offsets over a 
channel, in addition to the current fixed length transfers, the bufiter is 
defined to have a mode mask that can be set before commit time to describe the 
capabilities that will be used by this bufiter. This mask is not specified when 
the bufiter is created, but can be modified after creation in order to maintain 
source compatibility with the current 1.0 specification. The default mode of 
operation of the bufiter is unchanged. It is also the case that the default 
behavior likely has the lowest processor overhead and may have less network 
overhead because the additional modes require transferring additional buffer 
parameters along with the data.

Two new bufiter functions are provided:

MPIRT_BUFITER_SET_MODE(MPIRT_Bufiter bufiter, MPIRT_Bufiter_mode mode)
MPIRT_BUFITER_GET_MODE(MPIRT_Bufiter bufiter, MPIRT_Bufiter_mode *mode)

The mode is a bit mask, with the mode bits supported in any combination. A value 
of 0 for the mode specifies that the bufiter should behave according to the 1.0 
behavior. The mode bits are:

MPIRT_BUFITER_MODE_VARLEN: when set on the input bufiter for an output channel, 
the amount of data to be transferred is specified by the buffer object, not in 
the bufiter. When not set, the amount of data to be transferred is the value in 
the bufiter object. At commit time, it is verified that both ends of the channel 
have the same setting of MPIRT_BUFITER_MODE_VARLEN on the input bufiters.

MPIRT_BUFITER_MODE_VAROFFSET - when set, data transfer will start at an offset 
into the buffer. This offset is specified in the buffer object. The offset is 
only local - the other end of the channel doesn't have to match.

(Note to reviewers - these 2 bits can be used and still maintain a QoS. 
Additional mode bits could be defined to enable variable datatypes and unknown 
lengths, but this would involve transfer checking in the real-time portion of 
the application, and the only QoS that would make sense is "best effort".)

BUFFER OBJECT FUNCTIONS

Variable Length Transfers

The next 2 functions are provided to set/get the number of elements used
in the buffer:

MPIRT_BUFFER_SPECIFY_COUNT(MPIRT_Buffer buffer, int elements)
MPIRT_BUFFER_RETRIEVE_COUNT(MPIRT_Buffer buffer, int *elements)

MPIRT_BUFFER_SPECIFY_COUNT specifies the number of data elements to be 
transmitted on output channels of mode MPIRT_BUFITER_MODE_VARLEN. This length 
must be less than or equal to the busize specified in the bufiter. The default 
value for length is bufsize, so if MPIRT_BUFFER_SPECIFY_COUNT is not called, the 
entire buffer will be transmitted. For an input channel, an implicit 
MPIRT_BUFFER_SPECIFY_COUNT is called when the buffer is filled to specify the 
number of data elements actually received. MPIRT_BUFFER_RETRIEVE_COUNT can then 
be used to get the number of elements received. This behavior happens whether 
MPIRT_BUFITER_MODE_VARLEN is set or not. If (offset + count) is greater than 
the buffer's bufsize, MPIRT_ERR_BUFSPEC is returned when 
MPIRT_BUFFER_SPECIFY_COUNT is called, but the count is still set in the buffer. 
This call should not be used if the buffer is already in a bufiter - results 
are unpredictable if the buffer is in a bufiter.

MPIRT_BUFFER_RETRIEVE_COUNT retrieves the count value.

Variable Buffer Offsets

For some applications it would reduce the amount of code and number of buffer 
objects to be able to use the same buffer object to transfer data from 
different locations within a data buffer.

The 2 functions to support this are:

MPIRT_BUFFER_SPECIFY_OFFSET(MPIRT_Buffer buffer, int offset)
MPIRT_BUFFER_RETRIEVE_OFFSET(MPIRT_Buffer buffer, int *offset)

This buffer object value is not examined unless MPIRT_BUFITER_MODE_VAROFFSET is 
set in the bufiter. When enabled, data transfer starts at offset elements from 
the start of the buffer. The offset must be non-negative. If (offset + count) is 
greater than the buffer's bufsize, MPIRT_ERR_BUFSPEC is returned, but the offset 
is still set in the buffer. Notice that if VAROFFSET is being used, but VARLEN 
is not being used, MPIRT_ERR_BUFSPEC will always be returned. It is up to the 
user to make sure that the offset is valid. This call should not be used if the 
buffer is already in a bufiter - results are unpredictable if the buffer is in a 
bufiter.

MPIRT_ERR_BUFSPEC can be returned even though the buffer will end up in a valid 
state as a result of another SPECIFY call. This can be avoided by calling 
MPIRT_BUFFER_SPECIFY_COUNT and MPIRT_BUFFER_SPECIFY_OFFSET in the correct order. 
If the count is going to be reduced and the offset increased, change the count 
before the offset. If the count is going to increase and the offset is going to 
decrease, change the offset before the count. To avoid this error, the following 
function can be called which changes both parameters simultaneously:

MPIRT_BUFFER_SPECIFY_OFFSET_COUNT(MPIRT_Buffer buffer, int offset, int count)

For an output channel, data starting at offset for count elements is transmitted. 
For an input channel, only elements starting at offset is modified. The 
remainder of the data in the buffer is unmodified.

Note that the offsets used by the input and output ends of the channels do not 
have to match. In fact, it is allowed that only one end of the channel be using 
offset mode.

Discussion of BUFFER SIZES and DATATYPES in BUFITERS

The restrictions on the characteristics of buffer objects that may be added to a 
bufiter should be changed to the following:

The size, offset and datatype for buffers added to bufiters used in channel 
operations must be compatible with the requirements of that channel in order to 
satisfy the QoS requirements of that channel. The datatype of the buffer must 
match the datatype of the bufiter. The length does not have to match. Buffers 
added to the input bufiter of an input channel must have room for the maximum 
transfer as specified by the bufsize of the bufiter in order to guarantee that 
there will be enough space to handle any legal transfer. For an output channel, 
the amount of data to be transferred must be no more than specified in the 
bufsize parameter of the input bufiter. There are 2 cases depending on the 
setting of MPIRT_BUFITER_MODE_VARLEN. If not set, the buffer must be at least as 
large as the setting of bufsize in the bufiter, and only the first bufiter 
bufsize elements will be transferred (starting at offset if offset mode is 
enabled). If MPIRT_BUFITER_MODE_VARLEN is set, there isn't any restriction on the 
size (bufsize) of the buffer, but the actual amount of data (specified with 
MPIRT_BUFFER_SPECIFY_LENGTH() ) to be transferred must be no more than the 
bufiter bufsize.

NOTE TO IMPLEMENTORS: the various bufiter modes may add some fixed overhead to 
buffer processing, and it isn't known at commit time how much data will actually 
be transferred. However, the worst case data transfer is known because the 
bufsize parameter in the bufiter defines the longest allowed transfer over the 
channel. As a result, all channel QoS variations are allowed.

Note to reviewers: we need an error return for inserting an invalid buffer into 
a bufiter:

MPIRT_ERR_BUFSPEC - returned if the buffer is too small to accept the maximum 
data transfer for an input channel as specified in the input bufiter, or the 
requested transfer exceeds the maximum transfer size when in varlen mode.

Note to reviewers: do we need an error if a buffer with modified offset is used 
on a channel that ignores the offset? I propose NOT. It is NOT an error to 
insert a buffer with modified count into a channel that ignores the length.

A change to the bufiter description is required - buf size is now the MAXIMUM 
transfer size to be performed. Both ends of the channel must still match.