8.5.3. Flow-of-Control

Describe how allocation requests are flowed to the scheduler, scheduler attachment upon startup of either side, where response to allocation requests are returned and how they get back to the requestor, where session controls are received for allocation instantiation and how that is done, where session control is called to indicate all jobs complete in session.

Probably best to start with an overview of how things flow and then drill down into the respective steps.

Allocation requests have to be serviced by the scheduler. Since a requestor could attach directly to the scheduler (e.g., in the case of a tool submitting a session request) or to a daemon within the RM (who must then relay it to its system controller for forwarding to the scheduler), the scheduler must inform the RM of the allocation via the PMIx_Session_control API - i.e., the RM cannot guarantee its ability to intercept and process an allocation response to learn of a session that needs to be instantiated.

A scheduler attaches to the DVM master as a tool that declares itself the scheduler (PMIX_SERVER_SCHEDULER), naming its own namespace and rank. The master accepts that only from a tool running as root, as the user the DVM runs as, or as a user named by the prte_pmix_scheduler_uids MCA parameter - a comma-separated list of user names or numeric ids, for a scheduler that runs as a service user of its own. It forgets the scheduler when it disconnects, after which the DVM answers session and allocation requests itself again.

PRRTE’s role in the PMIx_Allocation_request flow is generally to pass the request on to the scheduler, and transport the reply back to the requestor.

When PRRTE passes a request on - to the scheduler, or to another DVM’s data server - the scheduler’s PMIx tells it that PRRTE is asking, and PRRTE says whom it asks for in PMIX_ORIGIN_USERID, PMIX_ORIGIN_GRPID and the other PMIX_ORIGIN_* attributes, beside PMIX_REQUESTOR. Whether to believe that is the receiving host’s decision. PRRTE, receiving a relayed request itself, believes it from a relay running as the same user it names, as root, or as the user the DVM runs as. Any other relay’s request is taken as the relay’s own.

Every request for new or extended resources (PMIX_ALLOC_NEW, PMIX_ALLOC_EXTEND) also says whom the allocation is for, as PMIX_ALLOC_UID and PMIX_ALLOC_GID. A requester may name its own user - root may name any user - and any group it belongs to; naming anything else is refused with PMIX_ERR_NO_PERMISSIONS before the request goes anywhere. Where the requester names neither, PRRTE adds its own user and group. The scheduler is to grant the allocation for exactly that user and group and return both in its answer; an answer that names another, or none, is an error, and PRRTE hands what was granted back. The allocation then belongs to that user and group - they are who may later extend or release it.

PMIX_ALLOC_ACTIVATE is the exception: PRRTE answers it itself, without involving any resource manager. It asks for a daemon on nodes the requestor already holds - an allocated node the DVM is not spanning, because a --host/--hostfile given at startup narrowed which nodes got one, or because a released reservation handed it back without one. Nothing is added to the allocation and no slot count changes, so there is nothing to ask a scheduler for and nothing a scheduler could refuse; for the same reason the request is served even where the scheduler owns the allocation, which is where a request for new resources cannot be. The nodes are named with PMIX_HOST and/or PMIX_HOSTFILE, in the same syntax the --activate command line option takes, and the request is answered when it is granted - the daemons themselves are reported by the PMIX_DVM_IS_READY event that completes any DVM size change.

Under Slurm, a client’s PMIX_ALLOC_EXTEND is answered once queued, with PMIX_OPERATION_IN_PROGRESS and the PMIX_ALLOC_ID. PMIX_DVM_IS_READY or PMIX_ERR_DVM_MOD reports the grant. Until then PMIX_ALLOC_REQ_CANCEL cancels it; after it, the cancel is answered PMIX_ERR_EXISTS with the PMIX_ALLOC_ID, and the allocation must be released. That PMIX_DVM_IS_READY, and PMIX_QUERY_ALLOC_PROPERTIES, carry PMIX_ALLOC_GRANT_TIME: when the scheduler granted it.

An allocation request can also arrive on a spawn, as PMIX_SPAWN_ALLOC: the value is an array of pmix_info_t holding the request’s directive (PMIX_ALLOC_REQ_DIRECTIVE) and the info a standalone request would have carried. PRRTE serves that request first, exactly as it would serve the standalone one, and only launches the job once it is granted - pointing the job at what it was given, so it maps onto those resources rather than onto everything else. The two failures are kept apart: an allocation that is refused fails the spawn with PMIX_ERR_JOB_ALLOC_FAILED and launches nothing, while a spawn that fails after the grant hands the allocation back before returning its own error. This saves a caller the “request, wait, read the id, spawn into it” sequence, and closes the window in that sequence where resources are held for a job that does not exist.

PRRTE only creates a session object as a result of a call from the scheduler via PMIx_Session_control. What it does with each directive of that API - and where it deviates from the letter of the attribute description, as it must for preemption - is documented in Session Control.