| 55 | | Therefore, the TH_TBL(K,P) thread table for a given process in a given clusters contains only the threads of P placed in cluster K. The set of all threads of a given process is defined by the union of all TH_TBL(K,P) for all active clusters K. |
| 56 | | To scan the set off all threads of a process P, ALMOS-MKH traverse the COPIES_LIST of all process_descriptors associated to P process. |
| | 55 | Therefore, the TH_TBL(K,P) thread table for a given process in a given cluster contains only the threads of P placed in cluster K. The set of all threads of a given process is defined by the union of all TH_TBL(K,P) for all active clusters K. |
| | 56 | To scan the set of all threads of a process P, ALMOS-MKH traverses the COPIES_LIST of all process_descriptors associated to P process. |
| 58 | | This implementation of ALMOS-MKH does not support thread migration: a thread is pinned on a given core in a given cluster. In the future process migration mechanism, all threads of given process in a given cluster can migrate to another cluster for load balancing. This mechanism is not implemented yet (february 2018), and will require to distinguish the kernel thread identifier (TRDID, that will be modified by a migration), and the user thread identifier (THREAD, that cannot be modified by a migration). In the current implementation, the user identifier (returned by the pthread_create() sys call), is identical to the kernel identifier. |
| | 58 | This implementation of ALMOS-MKH does not support thread migration: a thread is pinned on a given core in a given cluster. In the future process migration mechanism, all threads of given process in a given cluster can migrate to another cluster for load balancing. This mechanism is not implemented yet (february 2018), and will require to distinguish the kernel thread identifier (TRDID, that will be modified by a migration), and the user thread identifier (THREAD, that cannot be modified by a migration). In the current implementation, the user identifier (returned by the pthread_create() sys call) is identical to the kernel identifier. |
| 80 | | The new C process inherit from the parent process P the open files (FDT), and the memory image (VSL and GPT). These structures must be replicated in the new process descriptor. After a fork(), the C process can execute an exec() system call, that allocate a new memory image to the C process, but the new process can also continue to execute with the inherited memory image. For load balancing, ALMOS-MKH uses the DQDT to create the child process C on a different cluster from the parent cluster P, but the user application can also use the non-standard fork_place() system call to specify the target cluster. |
| | 80 | The new C process inherits (from the parent process P) the open files (FDT), and the memory image (VSL and GPT). These structures must be replicated in the new process descriptor. After a fork(), the C process can execute an exec() system call, that allocates a new memory image to the C process, but the new process can also continue to execute with the inherited memory image. For load balancing, ALMOS-MKH uses the DQDT to create the child process C on a different cluster from the parent cluster P, but the user application can also use the non-standard fork_place() system call to specify the target cluster. |
| 86 | | The fork() system call is the only method to create a new process. A thread of parent process P, running in a cluster X, executes the fork() system call to create a child process C on a remote cluster Y, that will become both the owner and the reference cluster for the C process. A new process descriptor, and a new thread descriptor are created and initialized in target cluster Y for the child process. |
| 87 | | The calling thread can run in any cluster. If the target cluster Y is different from the calling thread cluster X, the calling thread uses a RPC to ask the target cluster Y to do the work, because only the target cluster Y can allocate memory for the new process and thread descriptor. |
| | 86 | The fork() system call is the only method to create a new process. A thread of parent process P, running in a cluster X, executes the fork() system call to create a child process C on a remote cluster Y, that will become both the owner and the reference cluster for the C process. A new process descriptor and a new thread descriptor are created and initialized in target cluster Y for the child process. |
| | 87 | The calling thread can run in any cluster. If the target cluster Y is different from the calling thread cluster X, the calling thread uses a RPC to ask the target cluster Y to do the work, because only the target cluster Y can allocate memory for the new process and thread descriptor. |
| 90 | 90 | * for the '''DATA, MMAP, REMOTE''' vsegs (containing shared, non replicated data), all vsegs registered in the parent reference VSL(Z,P) are registered in the child reference VSL(Y,C), and all valid GPT entries in the reference parent GPT(Z,P) are copied in the child reference GPT(Y,C). For all pages, the WRITABLE flag is reset and the COW flag is set, in both (parent and child) GPTs. This require to update all corresponding entries in the parent GPT copies (in clusters other than the reference). |
| 91 | 91 | * for the '''STACK''' vsegs (that are private), only one vseg is registered in the child reference VSL(Y,C). This vseg contains the user stack of the user thread requesting the fork, running in cluster X. All valid GPT entries in the parent GPT(X,P) are copied in the child GPT(Y,C). For all pages, the WRITABLE flag is reset and the COW flag is set, in both (client and child) GPTs. |
| 92 | | * for the '''CODE''' vsegs (that must be replicated in all clusters containing a thread), all vsegs registered in the reference parent VSL(Z,P) are registered in the child reference VSL(Y,C), but the reference child GPT(Y,C) is not updated by the fork: It will be dynamically updated on demand in case of page fault. |
| 93 | | * for the '''FILE''' vsegs (containing shared memory mapped files), all vsegs registered in the reference parent VSL(Z,P) are registered in the child reference VSL(Y,C), and all valid entries registered in the reference parent GPT(Z,P) are copied in the reference child GPT(Y,C). The COW flag is not set for these shared data. |
| | 92 | * for the '''CODE''' vsegs (that must be replicated in all clusters containing a thread), all vsegs registered in the reference parent VSL(Z,P) are registered in the child reference VSL(Y,C), but the reference child GPT(Y,C) is not updated by the fork: It will be dynamically updated on demand in cases of page fault. |
| | 93 | * for the '''FILE''' vsegs (containing shared memory mapped files), all vsegs registered in the reference parent VSL(Z,P) are registered in the child reference VSL(Y,C), and all valid entries registered in the reference parent GPT(Z,P) are copied in the reference child GPT(Y,C). The COW flag is not set for these shared data. |
| 101 | | At the end of the fork(), cluster Y is both the owner cluster and the reference cluster for the new C process, that contains one single thread running in the Y cluster. |
| 102 | | All pages of DATA, REMOTE, and MMAP vsegs are marked ''Copy On Write'' in the child C process GPT (clusters Y), and in all copies of the parent P process GPT (all clusters containing a copy of P). |
| | 101 | At the end of the fork(), cluster Y is both the owner cluster and the reference cluster for the new C process, that contains one single thread running in the Y cluster. |
| | 102 | All pages of DATA, REMOTE, and MMAP vsegs are marked ''Copy On Write'' in the child C process GPT (cluster Y), and in all copies of the parent P process GPT (all clusters containing a copy of P). |
| 108 | | After a fork() system call, any thread of a process P can execute an exec() system call. This system call forces the P process to execute a new application, while keeping the same PID, the same parent process, the same open file descriptors, and the same environment variables. The existing P process descriptors (both the reference and the copies) and all associated threads will be destroyed. A new process descriptor and a new main thread descriptor are created in the reference cluster, and initialized from values found in the existing process descriptor, and from values contained in the .elf file defining the new application. The calling thread can run in any cluster. If the reference cluster Z for process P is different from the calling thread cluster X, the calling thread must use a RPC to ask the reference cluster Z to do the work. |
| | 108 | After a fork() system call, any thread of a process P can execute an exec() system call. This system call forces the P process to execute a new application, while keeping the same PID, the same parent process, the same open file descriptors, and the same environment variables. The existing P process descriptors (both the reference and the copies) and all associated threads will be destroyed. A new process descriptor and a new main thread descriptor are created in the reference cluster, and initialized from values found in the existing process descriptor, and from values contained in the .elf file defining the new application. The calling thread can run in any cluster. If the reference cluster Z for process P is different from the calling thread cluster X, the calling thread must use a RPC to ask the reference cluster Z to do the work. |
| 120 | | If the target cluster K' is different from the client cluster K, the cluster K send a RPC_THREAD_USER_CREATE request to cluster K'. The argument is a complete structure pthread_attr_t (defined in the ''thread.h'' file in ALMOS-MK), containing the PID, the function to execute and its arguments, and optionally, the target cluster and target core. This RPC should return the thread TRDID. |
| | 120 | If the target cluster K' is different from the client cluster K, the cluster K sends a RPC_THREAD_USER_CREATE request to cluster K'. The argument is a complete structure pthread_attr_t, containing the PID, the function to execute and its arguments, and optionally, the target cluster and target core. This RPC should return the thread TRDID. |
| 122 | | * If the target cluster K' does not contain a copy of the P process descriptor, the kernel K' creates a process descriptor copy from the reference P process descriptor, using a remote_memcpy(), and using the cluster_get_reference_process_from_pid() to get the extended pointer on reference cluster. It allocates memory for the associated structures GPT(M,P), VSL(M,P), FDT(M,P). These structures being used as read-only caches will be dynamically filled by the page faults. This new process descriptor is registered in the COPIES_LIST and in the LOCAL_LIST. |
| | 122 | * If the target cluster K' does not contain a copy of the P process descriptor, the kernel K' creates a process descriptor copy from the reference P process descriptor, using a remote_memcpy(), and using the [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/kernel/kern/cluster.c#L349 cluster_get_reference_process_from_pid()] to get the extended pointer on reference cluster. It allocates memory for the associated structures GPT(M,P), VSL(M,P), FDT(M,P). These structures, being used as read-only caches, will be dynamically filled by the page faults. This new process descriptor is registered in the COPIES_LIST and in the LOCAL_LIST. |
| 129 | | The unique method to destroy a thread is to call the '''thread_kill()''' function, that set |
| 130 | | The scheduler calls the ''thread_destroy()'' function that detach the thread from the scheduler, detach the thread from the local process descriptor, and releases the memory allocated to the thread descriptor. The '''thread_kill()''' function can be called by the target thread itself (for an exit), or by another thread (for a kill). |
| | 129 | The unique method to destroy a thread is to call the '''thread_kill()''' function, that sets the THREAD_FLAG_REQ_DELETE bit in the ''flags'' field of the target thread descriptor. The thread will be asynchronously deleted by the scheduler at the next scheduling point. |
| | 130 | The scheduler calls the ''thread_destroy()'' function that detaches the thread from the scheduler, detaches the thread from the local process descriptor, and releases the memory allocated to the thread descriptor. The '''thread_kill()''' function can be called by the target thread itself (for an exit), or by another thread (for a kill). |
| 146 | | The thread destruction is more complex if the target thread T is running in ATTACHED mode, because another joining thread J, executing the ''pthread_join()'' system call, must be informed of the termination of thread T. As the ''thread_kill()'' function, executed by the killer thread K, and the ''sys_thread_join()'', executed by the joining thread J, can be executed in any order, this requires a "rendez-vous": The first arrived thread blocks and deschedules. It will be unblocked by the other thread. |
| | 146 | The thread destruction is more complex if the target thread T is running in ATTACHED mode, because another joining thread J, executing the ''pthread_join()'' system call, must be informed of the termination of thread T. As the ''thread_kill()'' function (executed by the killer thread K) and the ''sys_thread_join()'' (executed by the joining thread J) can be executed in any order, this requires a "rendez-vous": The first arrived thread blocks and deschedules. It will be unblocked by the other thread. |
| 154 | | * If the FLAG_JOIN_DONE is set, the J thread arrived first and is blocked on the BLOCKED_JOIN condition: the K thread unblock |
| 155 | | * If the FLAG_JOIN_DONE is not set, the K thread arrived first: the K thread blocks the K thread on the BLOCKED_JOIN condition, set the FLAG_KILL_DONE in the T thread, register the killer thread extended pointer in the T thread ''join_xp'' field, releases the ''join_lock'' in the T thread, and deschedules. It completes the T thread destruction as described in the detached case when it is unblocked by the J thread. |
| 156 | | * The J thread test |
| 157 | | * If the FLAG_KILL_DONE is set, the K thread arrived first and is blocked on the BLOCKED_JOIN condition: the J thread unblocks the killer thread, reset the FLAF_KILL_DONE in the T thread, releases the "join_lock" in T thread, and returns. |
| 158 | | * If the FLAG_KILL_DONE is not set, the J thread arrived first: the J thread register |
| | 154 | * If the FLAG_JOIN_DONE is set, the J thread arrived first and is blocked on the BLOCKED_JOIN condition: the K thread unblocks the J thread from the BLOCKED_JOIN condition (using the ''join_xp'' field in the T thread), reset the JOIN_DONE flag in T thread, releases the ''join_lock'' in T thread, and completes the T thread destruction as described in the detached case. |
| | 155 | * If the FLAG_JOIN_DONE is not set, the K thread arrived first: the K thread blocks the K thread on the BLOCKED_JOIN condition, sets the FLAG_KILL_DONE in the T thread, registers the killer thread extended pointer in the T thread ''join_xp'' field, releases the ''join_lock'' in the T thread, and deschedules. It completes the T thread destruction as described in the detached case when it is unblocked by the J thread. |
| | 156 | * The J thread tests the FLAG_KILL_DONE in the T thread descriptor: |
| | 157 | * If the FLAG_KILL_DONE is set, the K thread arrived first and is blocked on the BLOCKED_JOIN condition: the J thread unblocks the killer thread, resets the FLAG_KILL_DONE in the T thread, releases the "join_lock" in T thread, and returns. |
| | 158 | * If the FLAG_KILL_DONE is not set, the J thread arrived first: the J thread registers its extended pointer in the T thread "join_xp" field, set the FLAG_JOIN_DONE in the T thread, sets the BLOCKED_EXIT bit in the J thread, releases the "join_lock" in the T thread, and deschedules. It simply returns when it is unblocked by the K thread. |
| 168 | | The process destruction in the owner cluster is more complex, because the child process destruction must be reported to the parent process when the main thread of the parent process executes the blocking [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/kernel/syscalls/sys_wait.c sys_wait()] system call (in the parent owner cluster). Therefore, the child process in owner cluster cannot be destroyed before the parent calls the sys_wait() function. As the '''sys_wait()''' function, and the '''sys_kill()''' or '''sys_exit()''' function are executed by different threads running in different clusters, this requires a parent/child synchronization. To keep a process descriptor in ''zombi'' state after a sys-kill() or sys_exit(), the main thread (i.e. thread 0 in process owner cluster) is not deleted until the sys_wait() syscall is executed by the parent process main thread. This synchronization uses the '''term_state''' field in process descriptor, that contains the following informations : |
| 169 | | * the PROCESS_FLAG_KILL indicates that a KILL request has been received by the child; |
| 170 | | * the PROCESS_FLAG_EXIT indicates that an EXIT request has been made by the child; |
| 171 | | * the PROCESS_FLAG_BLOCK indicates that a SIGSTOP signal has been received by the child; |
| 172 | | * the PROCESS_FLAG_WAIT flag indicates that a WAIT request from parent has been received by the child. |
| 173 | | * moreover, for an exit(), the exit() argument value is registered in this ''term_state'' field. |
| | 168 | The process destruction in the owner cluster is more complex, because the child process destruction must be reported to the parent process when the main thread of the parent process executes the blocking [https://www-soc.lip6.fr/trac/almos-mkh/browser/trunk/kernel/syscalls/sys_wait.c sys_wait()] system call (in the parent owner cluster). Therefore, the child process in owner cluster cannot be destroyed before the parent calls the sys_wait() function. As the '''sys_wait()''' and the '''sys_kill()''' (or '''sys_exit()''') functions are executed by different threads running in different clusters, this requires a parent/child synchronization. To keep a process descriptor in ''zombie'' state after a sys-kill() or sys_exit(), the main thread (i.e. thread 0 in process owner cluster) is not deleted until the sys_wait() syscall is executed by the parent process main thread. This synchronization uses the '''term_state''' field in process descriptor, that contains the following information : |
| | 169 | * The PROCESS_FLAG_KILL indicates that a KILL request has been received by the child; |
| | 170 | * The PROCESS_FLAG_EXIT indicates that an EXIT request has been made by the child; |
| | 171 | * The PROCESS_FLAG_BLOCK indicates that a SIGSTOP signal has been received by the child; |
| | 172 | * The PROCESS_FLAG_WAIT flag indicates that a WAIT request from parent has been received by the child. |
| | 173 | * Moreover, for an exit(), the exit() argument value is registered in this ''term_state'' field. |
| 175 | | The actual deletion of the child owner process descriptor and child main thread are done by the sys_wait() function, executed by the parent main thread (i.e. thread 0 in parent owner cluster). This sys_wait() function executes an infinite loop. At each iteration the parent main thread scan all children owner descriptors. When it detects that one child terminated, it set the PROCESS_FLAG_WAIT in child owner process descriptor, set the THREAD_FLAG_DELETE in the child main thread, and returns to report the child termination state to parent process. It is the responsibility of the parent process to re-enter the sys_wait() syscall for the other children. When the parent process does not detect a terminated child at the end of an iteration, it deschedules without blocking. |
| | 175 | The actual deletion of the child owner process descriptor and child main thread are done by the sys_wait() function, executed by the parent main thread (i.e. thread 0 in parent owner cluster). This sys_wait() function executes an infinite loop. At each iteration, the parent main thread scans all children owner descriptors. When it detects that one child terminated, it sets the PROCESS_FLAG_WAIT in child owner process descriptor, sets the THREAD_FLAG_DELETE in the child main thread, and returns to report the child termination state to parent process. It is the responsibility of the parent process to re-enter the sys_wait() syscall for the other children. When the parent process does not detect a terminated child at the end of an iteration, it deschedules without blocking. |
| 186 | | 1. The sys_exit() function block the main thread, and set the PROCESS_TERM_EXIT flag in owner process descriptor to ask the parent process (sys_wait function) to mark this main thread for delete, and deschedules. The calling thread will be destroyed at the next scheduling point. |
| 187 | | 1. The main thread, and the owner process descriptor on one hand, the calling thread and the associated process will be destroyed by the scheduler at the next scheduling point. |
| | 186 | 1. The sys_exit() function blocks the main thread, and sets the PROCESS_TERM_EXIT flag in owner process descriptor to ask the parent process (sys_wait function) to mark this main thread for delete, and deschedules. The calling thread will be destroyed at the next scheduling point. |
| | 187 | 1. The main thread, and the owner process descriptor on one hand, the calling thread and the associated process will be destroyed by the scheduler at the next scheduling point. |
| 193 | | 1. The sys_kill() syscali must be executed by the main thread of the target process, OR by any thread of another process than the target process. |
| 194 | | 1. The sys_kill() function calls the process_sigaction() function that send a multicast, parallel and non blocking RPC to all clusters containing at least one thread of the target process, to block all process threads, but the main thread. This function returns only when all threads (but the main) are blocked and descheduled. |
| 195 | | 1. The sys_kill() function calls again the process_sigaction() function that send |
| 196 | | 1. The sys_kill() function set |
| | 193 | 1. The sys_kill() syscall must be executed by the main thread of the target process, OR by any thread of another process than the target process. |
| | 194 | 1. The sys_kill() function calls the process_sigaction() function that sends a multicast, parallel and non blocking RPC to all clusters containing at least one thread of the target process, to block all process threads, except for the main thread. This function returns only when all threads (but the main) are blocked and descheduled. |
| | 195 | 1. The sys_kill() function calls again the process_sigaction() function that sends another multicast, parallel and non blocking RPC to the same clusters, to mark for delete all process threads, but the main thread. The marked threads will be actually destroyed by the scheduler at the next scheduling point. The target process descriptor copies are actually destroyed by the scheduler when the last thread in remote cluster is destroyed. |
| | 196 | 1. The sys_kill() function sets the PROCESS_TERM_KILL flag in the target process descriptor in owner cluster to ask synchronize with its parent process, and returns. |
