Changes between Version 54 and Version 55 of replication_distribution


Ignore:
Timestamp:
Dec 9, 2019, 8:29:52 PM (5 years ago)
Author:
alain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • replication_distribution

    v54 v55  
    11= Data replication & distribution policy =
     2
     3alain.greiner@lip6.fr
    24
    35[[PageOutline]]
     
    5153For more details on the VMM implementation, the API is defined in the [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/kernel/mm/vmm.h almos_mkh/kernel/mm/vmm.h] and [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/kernel/mm/vmm.c almos-mkh/kernel/mm/vmm.c] files.
    5254
    53 == __2. User segments__  ==
     55== __2. User vsegs__  ==
    5456
    5557This section describes the six types of user virtual segments and the associated replication / distribution policy defined and implemented by almost-mkh:
     
    9294
    9395
    94 == __ 3. kernel segments__==
     96== __ 3. kernel vsegs__==
    9597
    9698For any process descriptor P in a cluster K, the VMM(P,K) contains not only the user vsegs defined above, but also the kernel vsegs, because all user theads can make system calls, that must access both the kernel instructions and the kernel data structures, and this requires address translation. This section describes the four types of kernel virtual segments defined by almost-mkh.
     
    98100=== 3.1. KCODE vsegs ===
    99101
    100 The KCODE vseg contains the kernel code defined in the ''kernel.elf'' file.  Almos-mkh creates one KCODE vseg in each cluster K, to avoid contention. It is a ''private'' vseg, that is accessed only by the threads running in cluster K. It can be an user thread executing a syscall, or it can be a specialized kernel thread (such as an IDLE thread, a DEV thread, or a RPC thread). In each cluster K, the KCODE vseg is registered in the VMM(0,K) associated to the kernel ''process_zero'', that contains all kernel threads, and in the VMM(P,K) of each user process P that has at least one thread running in cluster K. This vseg uses only big pages, that are mapped by the kernel_init function (no on demand paging for this vseg).
     102A '''KCODE''' vseg contains the kernel code defined in the ''kernel.elf'' file.  To avoid contention and improve locality, almos-mkh replicates this code in all clusters. This code has already been copied in all clusters by the bootloader. In each cluster K, and for all process P in cluster K (including the kernel process_zero), almos-mkh registers the KCODE vseg in all VSL(P,K), and map it to the local copy in all the GPT(P,K). This vseg uses only big pages, and there is no on-demand paging for this type of vseg. With this local mapping all access to the virtual instruction addresses will be simply translated by the MMU to the local physical address.
     103
     104'''WARNING''' : there is only one vseg defined in the ''kernel.elf'' file, but there is as many KCODE vsegs as the number of clusters. All these vsegs have the same virtual base address and the same size. but the physical adresses (defined in the GPTs) depend on the cluster, because we want to access the local copy.
     105This is not a problem because A KCODE vseg is a ''private'' vseg, that is accessed only by local threads.
    101106
    102107=== 3.2. KDATA vsegs ===
    103108
    104 This '''public''' vseg contains the global data,  statically allocated at compilation time. This vseg is also replicated in all clusters. The values initially contained in these KDATA vsegs are identical, as they are defined in the ''kernel.elf'' file. But they are not read-only,  and can evolve differently in different clusters.  As the KDATA vsegs are replicated in all clusters, most accesses to a KDATA segment are expected to be done by local threads. These local accesses can use the normal pointers in virtual kernel space.
    105 
    106 But there is only one vseg defined in the ''kernel.elf'' file, and there is as many KDATA segments as the number of clusters. Even if most accesses are local, a thread running in cluster K must be able to access a global variable stored in another cluster X, or to send a request to another kernel instance in cluster X, or to scan a globally distributed structure, such as the DQDT or the VFS. To support this cooperation between kernel instances, almos-mkh defines the ''remote_load( cxy , ptr )'' and ''remote_store( cxy , ptr ) functions, where ''ptr'' is a normal pointer in kernel virtual space on a variable stored in the KDATA vseg, and ''cxy'' is the remote cluster identifier. Notice that a given global variable is now identified by and extended pointer ''XPTR( cry , ptr )''. With these remote access primitives, any kernel instance in cluster K can access any global variable in any cluster.
     109A '''KDATA''' vseg contains the kernel global data,  statically allocated at compilation time, and defined in the ''kernel.elf'' file. To avoid contention and improve locality, almos-mkh replicates the KDATA vseg in all clusters. The corresponding data have already been copied in all clusters.  As a physical copy of the KDATA vseg is available in any cluster K, almos-mkh can register this vseg in all VSL(P,K), and map it to this local copy in all GPT(P,K). With this local mapping we expect that most accesses to any KDATA segment will be done by a local thread.
     110
     111
     112'''WARNING''' : there is only one vseg defined in the ''kernel.elf'' file, and there is as many KDATA vsegs as the number of clusters. All these vsegs have the same virtual base address and the same size, but the physical addresses (defined in the  GPTs), depends on the cluster, because we generally want to access the local copy.
     113This is a problem, because there is two  big differences between the KCODE and the KDATA vsegs :
     1141. The values contained in the N KDATA vsegs are initially identical, as they are all defined by the same ''kernel.elf'' file. But they are not read-only,  and can evolve differently in different clusters.
     1151. The N KDATA vsegs are ''public'', and define an addressable storage space N times larger than one single KDATA vseg.  Even if most accesses are local, a thread running in cluster K must be able to access a global variable stored in another cluster X, or to send a request to another kernel instance in cluster X, or to scan a globally distributed structure, such as the DQDT or the VFS.
     116To support this inter-cluster kernel-to-kernel communication, almos-mkh defines the ''remote_load( cxy , ptr )'' and ''remote_store( cxy , ptr ) functions, where ''ptr'' is a normal pointer (in kernel virtual space) on a variable stored in the KDATA vseg, and ''cxy'' is the remote cluster identifier. Notice that a given global variable is now identified by and extended pointer ''XPTR( cry , ptr )''. With these remote access primitives, any kernel instance in cluster K can access any global variable in any cluster. Notice that local accesses can use the normal pointers in virtual kernel space, as the virtual adresses will be simply translated by the MMU to the local physical address.
     117
     118In other words, almost-mkh clearly distinguish the ''local accesses'', that can use standard pointers, from the ''remote access'' that '''must''' extended pointers.
     119This can be seen as a bad constraint, but it can also help to imp to improve the locality, and to identify (and remove) the contention.
     120
     121The remote_access primitives API is defined in the  [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/hal/generic/hal_remote.h  almos_mkh/hal/generic/hal_remote.h] file.
    107122
    108123=== 3.3. KHEAP vsegs ===
    109124
    110 Beside the statically allocated global variables, a large number of kernel structures, such as the user ''process'' descriptors, the ''thread'' descriptors, the ''vseg'' descriptors, the ''file'' descriptors, etc. are dynamically descriptors
     125The '''KHEAP''' vsegs are used to store dynamically allocated of kernel structures, such as the user ''process'' descriptors, the ''thread'' descriptors, the ''vseg'' descriptors, the ''file'' descriptors, etc. These structure can be requested by any thread running in any cluster, and are defined as global variables, that can be accessed by any thread. To avoid contention and improve locality, almos-mkh build a ''physically distributed  kernel heap''. In each cluster K, almos-mkh  can register this HEAP vseg in all VSL(P,K), and map it in all the GPT(P,K). with is local mapping, a kernel structure requested by a thread running in any cluster will always be allocated in the local physical memory.
     126
     127WARNING : To unify the access to remote data (i.e. data stored in a remote cluster), almos-mkh use the same policy for KHEAP and KDATA vsegs: all KHEAP segments have the same virtual base address. The local accesses to the locally allocated kernel structures can use normal pointers that will be translated by the MMU to local physical adresses. The remote access to remotely allocated kernel structures must use the ''remote_load()'' and ''remote_store()'' functions and handle extended pointers.
    111128
    112129=== 3.4. KDEV vsegs ===
    113130
    114 
    115 == __4. Physical mapping of kernel vsegs__ ==
    116 
    117 The implementation of these remote access functions depends on the target architecture.
     131Finally the '''KDEV''' vsegs are associated to the peripheral. There is one KDEV vseg per chdev (i.e. per channel device.
     132
     133=== 3.5. Summary ===
     134
     135
     136
     137== __4. Address Translation for kernel vsegs__ ==
     138
     139The implementation of the remote access functions introduced in section 3 depends on the target architecture.
    118140
    119141
     
    121143the contentglobal data. ALMOS-MK creates one DATA vseg in each vseg, that is registered in the reference VSL(P,KREF) when the process P is created in reference cluster KREF.  In the other clusters K, the DATA vseg is registered  in VSL(P,K) when a page fault is signaled by a thread of P running in cluster K. To avoid contention, this vseg is physically distributed on all clusters, with a page granularity. For each page, the physical mapping is defined by the LSB bits of the page VPN.* The read-only segment containing the user code is replicated in all clusters where there is at least one thread using it.
    122144
    123 The remote_access primitives are defined in the  [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/hal/generic/hal_remote.h  almos_mkh/hal/generic/hal_remote.h] file.
    124145
    125146The TSAR implementation is available in the  [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/hal/tsar_mips32/core/hal_remote.c  almos_mkh/hal/tsar_mips32/core.hal_remote.c] file.
     
    129150
    130151As the TSAR architecture uses 32 bits cores, to reduce the power consumption, the virtual space is much  smaller (4 Gbytes) than the physical space.
    131 Even if
     152
     153But the TSAR architecturethe MIPS32 core used BY the TSAR architecture HAS TWO NON STANDARDdefines another, non-standard, hardware mechanism to build a 40 bits physical address from a 32 bits virtual address. build  an al
     154
     155This mechanism can bu used
    132156
    133157To enforce locality, there is one KDATA segment per cluster, containing a copy of all global variables statically allocated at compilation time. On the other hand, all structures dynamically allocated by the kernel (to create a new process descriptor, a new thread descriptor, a new file descriptor, etc.) are allocated in the KHEAP segment of the target cluster, and will be mainly handled by a kernel instance running in this same kernel. Therefore, most kernel memory accesses expected to be local.