Changes between Version 29 and Version 30 of rpc_implementation


Ignore:
Timestamp:
Apr 7, 2018, 3:09:31 PM (6 years ago)
Author:
alain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • rpc_implementation

    v29 v30  
    1515 * There is one kernel instance in each cluster containing at least one core, one interrupt controler, and one physical memory bank.
    1616
    17 == 2) Unique access point in cluster ==
     17== 2) Server Core Selection ==
     18
     19Any client thread T running in any cluster K can send an PRC request to any cluster K'. In order to share the working load associated with RPC handling, each core in server cluster K'
     20has a private RPC requests queue, where the client threads must register its RPC request. In principle, the client thread T running on the client core [i] select the waiting queue  queue [i] in server cluster K'. If it is not possible (when the number of cores in cluster K' is smaller than the number of cores in client cluster), the selected server core in server is core [0].
    1821
    1922ALMOS-MKH replicates the KDATA segment (containing the kernel global variables) in all clusters, and uses the same LPA (Local Physical Address) for the KDATA segment base.
    20 Therefore, in two different clusters, a given global variable, identified by its LPA can have different values. This feature is used by by ALMOS-MKH to allow a kernel instance in a cluster K to access the global variables of another kernel instance in cluster K', building a physical address by concatenation of a given LPA with the CXY cluster identifier for the target cluster K'.
     23Therefore, in two different clusters, a given global variable, identified by its LPA can have different values. This feature is used by by ALMOS-MKH to allow a client thread in cluster K to access a global variable in a server cluster K', building a physical address by concatenation of a given LPA with the CXY cluster identifier for the server cluster K'.
    2124
    22 ALMOS-MKH defines a software RPC_FIFO, implemented as a global variable in the KDATA segment. This RPC_FIFO, is a member of the "cluster_manager" structure, that is replicated in all clusters containing a kernel instance. It is used by client threads to register RPC requests to a given target cluster K.
     25For each core [i] in a cluster K, ALMOS-MKH implement the RPC requests queue as a software RPC_FIFO[i,k]], implemented as a global variable in the KDATA segment. More precisely, each RPC_FIFO[i] has the type ''remote_fifo_t'', and is a member of the "cluster_manager" structure of cluster [k].
    2326
    24 This RPC_FIFO has a large number (N) of writers, an a small number (M) of readers:
    25  * N is the number of client threads (practically unbounded). A client thread can execute in any cluster, and can send a RPC request to any target cluster K. To synchronize these multiple client threads, the RPC FIFO implements a "ticket based" policy, defining a ''first arrived / first served'' priority to register a new request into FIFO.
    26  * M is the number of server threads in a given cluster, (bounded by the CONFIG_RPC_THREAD_MAX parameter). The RPC server threads are dynamically created in cluster K to handle the RPC requests. To synchronize these multiple  server threads (called RPC threads), the RPC FIFO implements a "light lock", that is a non blocking lock: only one RPC thread at a given time can take the lock and become the FIFO owner. This FIFO owner is in charge of handling pending RPC requests. When the light lock is taken by RPC thread T, another RPC thread T' failing to take the lock simply returns to IDLE state.
     27This RPC_FIFO has been designed to support a large number (N) of concurrent writers, an a small number (M) of readers:
     28 * N is the number of client threads (practically unbounded). A client thread can execute in any cluster, and can send a RPC request to any target cluster K. To synchronize these multiple client threads, the RPC FIFO implements a "ticket based" policy, defining a ''first arrived / first served'' priority to register a new request into a given RPC_FIFO[i,k].
     29 * M is the number of server threads in charge of handling RPC requests stored in a given RPC_FIFO[i,k]. M is bounded by the CONFIG_RPC_THREAD_MAX parameter. For each PRC_FIFO[i,k], it can exist several server threads, because we must avoid the ''head-of-line" blocking phenomenon, when a given server thread handling a given RPC is blocked on a given resource. To synchronize these multiple  server threads, the RPC FIFO implements a "light lock", that is a non blocking lock: only one RPC thread at a given time can take the lock and become the FIFO owner. When the light lock is taken by RPC thread T, another RPC thread T' failing to take the lock simply returns to IDLE state.
    2730 
    28 == 3) Client blocking policy and synchronization ==
     31== 3) Client / Server Synchronization ==
    2932
    30 All RPCs are blocking for the client thread: the client thread register its request in the target RPC FIFO, blocks on the specific THREAD_BLOCKED_RPC condition, and deschedules. The client thread is unblocked by the RPC server thread, when the RPC is completed.
     33All RPC requests are blocking for the client thread: the client thread register its request in the target RPC_FIFO, blocks on the THREAD_BLOCKED_RPC condition, and deschedules. The client thread is unblocked by the RPC server thread, when the RPC is completed.
    3134
    32 In order to reduce the RPC latency for the client thread, ALMOS-MKH use IPIs (Inter-Processor Interrupts). An IPI forces the target core to make a scheduling. The client thread select a core in the target cluster, and send an IPI to this selected server core. When no RPC thread is active on the target cluster, the selected core will activate (or create) a new RPC thread and execute it. When an RPC thread is already active, the IPI forces a scheduling point on the target core, but no new RPC thread is activated.
     35In order to reduce the RPC latency, ALMOS-MKH use IPIs (Inter-Processor Interrupts). The client thread select a core in the server cluster, and send an IPI to the selected server core. An IPI forces the target core to make a scheduling. This reduces the RPC latency when no RPC thread is active for the server core, because the RPC threads have the highest scheduling priority. When no RPC thread is active for this core, the selected core will activate (or create) a new RPC thread and execute it. When an RPC thread is already active, the IPI forces a scheduling point on the target core, but no new RPC thread is activated (or created).
    3336
    34 To distribute the load associated to RPC handling on all cores in a given target cluster, ALMOS-MHH implement the following policy: The client thread select in target cluster the core that has the same local  index as the core running the client thread in the client cluster. If this is not possible (when the number of cores in the server cluster is smaller than the
    35 number of cores in the client cluster), ALMOS-MKH select the core 0.
    36 
    37 When the RPC server thread completes, it uses a remote access to unblock the client thread, and send and IPI to the client core to force a scheduling. This reduces the RPC latency, because the RPC threads are kernel thread that have the highest scheduling priority. an IPI to the RPC thread completing an RPC request send an IPI to force a scheduling on the core running the client thread.
     37Similarly, when the RPC server thread completes, it uses a remote access to unblock the client thread, and send and IPI to the client core to force a scheduling on the client core.
    3838
    3939== 4) Parallel RPC requests handling ==
    4040
    41 Each cluster contains a pool of RPC threads, in charge of RPC requests handling. As the scheduler takes into account the thread type, it can take into account the RPC_FIFO state for the RPC threads. AN RPC thread is not runnable when the RPC_FIFO is empty. When the RPC_FIFO is not empty, only one RPC thread has the FIFO ownership and can consume RPC requests from the FIFO.
     41As explained above, a private pool of RPC threads is associated to each RPC_FIFO[i,k]. These RPC server threads are dynamically created when required.
    4242
    43 Nevertheless, ALMOS-MKH supports several RPC threads (up to CONFIG_RPC_THREAD_MAX per cluster), because a given RPC thread T handling a given request can block, waiting for a shared resource, such as a peripheral. In that case, the blocked RPC thread T releases the FIFO ownership before blocking and descheduling. This RPC thread T will complete the current RPC request when the blocking condition is solved, and the thread T is rescheduled. If the RPC FIFO is not empty, another RPC thread T' will be scheduled to handle the pending RPC requests. If all existing RPC threads are blocked, a new RPC thread is dynamically created.
     43At any time, only one RPC thread has the FIFO ownership and can consume RPC requests from the FIFO. Nevertheless, ALMOS-MKH supports supports several RPC server threads per RPC_FIFO because a given RPC thread T handling a given request can block, waiting for a shared resource, such as a peripheral. In that case, the blocked RPC thread T releases the FIFO ownership before blocking and descheduling. This RPC thread T will complete the current RPC request when the blocking condition is solved, and the thread T is rescheduled. If the RPC FIFO is not empty, another RPC thread T' will be scheduled to handle the pending RPC requests. If all existing RPC threads are blocked, a new RPC thread is dynamically created.
    4444
    45 Therefore, it can exist N simultaneously active RPC threads: the running one is the FIFO owner, and the (N-1) others are blocked on a wait condition.
    46 
     45Therefore, it can exist for each RPC_FIFO[i,k] a variable number X of active RPC threads: the running one is the FIFO owner, and the (X-1) others are blocked on a wait condition.
     46This number X can temporarily exceed the CONFIG_RPC_THREAD_MAX value, but the but the exceeding threads are destroyed when the temporary overload is solved.
    4747
    4848== 5) RPC request format ==