Cache coherence protocol

1. General Principles

This section describes the DHCCP protocol (Distributed Hybrid Cache Coherence Protocol), implemented by the TSAR architecture. For scalability purposes, the TSAR architecture implement a “Directory Based” cache coherence policy. From a conceptual point of view, the coherence protocol is supported by a Global Directory located in the memory controller : This Global Directory stores the status of each cache line replicated in at least one L1 cache of the TSAR architecture.

The main goal being the protocol scalability, the L1 caches implement a WRITE-THROUGH policy. The coherence protocol is much simpler than the MESI protocol used in most architectures implementing a WRITE_BACK policy. With a WRITE-THROUGH policy, the main memory contains always the most recent value of a cache line, and there is NO exclusive ownership state for a L1 cache.

The basic mechanism is the following : when the memory controller receives a WRITE request for a given cache line, he must send an UPDATE or INVAL request to all L1 caches containing a copy (but the writer). The write request is acknowledged only when all UPDATE or INVAL transactions are completed.

In the TSAR architecture, the memory controller is distributed, as it is implemented by the distributed memory caches (one per cluster). Therefore, the global directory itself is distributed. The memory cache being inclusive: a cache line L that is present in at least one L1 cache must be present in the corresponding memory cache cache (in the home cluster). With this property, the Global Directory can be implemented as an extension of the memory cache directory.

In case of MISS, the memory cache controller must evict a victim line to bring in the missing line. In order to maintain the inclusive property, all copies of the evicted cache line in L1 caches must be invalidated. To do it, the memory cache controller must send invalidate requests to all L1 caches containing a copy.

The TSAR architecture wants to guaranty the cache coherence by hardware, for both the data and instruction L1 caches. The modifications of shared data are very frequent events, but the number of copies is generally not very high. The modifications of shared code are very rare events (self modifying code, or dynamic libraries), but the number of replicated copies can be very large ( the exception handler, or the libc are generally replicated in all L1 caches ). Reflecting the different behaviour of data & instruction caches, the "hybrid" cache coherence protocol DHCCP defines two different strategies, depending on the number of copies :

• MULTICAST_UPDATE : When the number of copies is smaller than the DHCCP threshold, the memory cache controller registers the locations of all the copies, and send a multicast_update transaction to each concerned L1 cache in case of modification.
• BROADCAST_INVAL : When the number of copies is larger than the DHCCP threshold, the memory cache controller registers only the number of copies (without localization) and send a broadcast_invalidate transaction to all L1 caches in case of modication.

2. Types of transaction

Three types of transactions, have been identified :

• Direct transactions : READ / WRITE / LL / SC
• Coherence transactions : MULTI_UPDATE / MULTI_INVAL / BROADCAST_INVAL / CLEANUP
• External transactions : PUT / GET

For dead-lock prevention, these three types of transaction must be transported on three (virtually or physically) separated networks.

As a general rule, all these transactions respect the VCI advanced packet format, and there is one response packet for each command packet : For a burst transaction, a READ command packet contains one single flit, and the corresponding READ response packet contains N flits. Symmetrically, a WRITE command packet contains N flits, and the corresponding WRITE response packet contains one single flit.

There is one exception : For a BROADCAST_INVAL transaction, the initiator sends one single flit VCI packet, but receives several single flit VCI response packets.

2.1 Direct transactions

These transactions are initiated by a processor (actually the L1 cache controller), or by another initiator (an I/O peripheral or hardware coprocessor with a DMA capability). This initiator can be located in any cluster. For those transactions, the target is a memory cache controller, acting as a physical memory bank, or another VCI target peripheral. This target can be located in any cluster.

The L1 cache controller can issue several simultaneous VCI transactions, that must be distinguished by the VCI TRDID field value. The VCI PKTID field is also used to distinguish transaction types.

• A READ transaction can be a single word request (in case of uncached access), or a burst, corresponding to a complete cache line (16 words). A READ burst transaction initiated by any DMA controller must respect the same 16 words cache line format. For all READ transaction, the VCI command packet contains one single VCI flit. The VCI CMD field contains the VCI_READ code. The VCI PLEN field is used to define the burst length (number of bytes). A READ transaction has a type, encoded in the three LSB bits of the VCI PKTID field (the MSB bit is ignored).

PKTID for Uncached Data	X000
PKTID for Cached Data	X001
PKTID for Uncached Instruction	X010
PKTID for Cached Instruction	X011

• A WRITE transaction can be a single word request or a variable length burst request. In case of burst, the VCI command packet contains at most 8 VCI flits, with consecutive addresses. All words belong to the same half cache line, and the VCI BE field can have different values for each flit (including the zero value). The VCI response packet contains one VCI flit. A WRITE burst transaction initiated by any DMA controller must respect the same 8 aligned words constraint. The VCI CMD field contains the VCI_WRITE code. The PKTID field contains the TYPE_WRITE code (X100, MSB bit is ignored). The TRDID field is used to code the slot index in the write buffer. If the modified cache line is replicated in one or several other L1 caches, the memory cache must guaranty that all copies have been updated or invalidated before the WRITE transaction is acknowledged.

TRDID for Write burst	wbuf_index
PKTID for Write burst	X100

• The TSAR architecture supports the LL/SC mechanism for atomic operations (see AtomicOperations).

For both a LL (Linked Load) or a SC (Store Conditionnal) transaction, the VCI command packet and the VCI response packet contain one single VCI flit. The VCI CMD field must contain the VCI_LINKED_READ value (resp. VCI_STORE_CONDITIONNAL value). The VCI TRDID field takes the 0 value for all LL or SC packets.

2.2 Coherence transactions

For each cache line stored in the memory cache, the memory cache implements a Registration Table that contain the copies replicated in the L1 caches. Each entry in this Registration Table contains the SRCID of the L1 cache that contains a copy, as well as the type of the copy (instruction/data). When the same cache line is replicated in both the instruction cache and the data cache of a processor, this defines two separated entries in the Registration Table. When the number copies for a given cache line L exceeds the DHCCP threshold, the corresponding Registration Table is flushed, and the memory cache registers only the number of copies.

The coherence transactions use a logically separated coherence network, implementing a separated address space. All these transactions are write transactions.

• A MULTI_UPDATE transaction is a multi-cast transaction sent by the memory cache controller when it receives a WRITE request to a replicated cache line and the number of copies does not exceeds the DHCCP threshold. It sends as many VCI transactions as the number of registered copies (but the writer). The VCI command packet contains (N+2) flits. The VCI ADDRESS field is constant and contains the address of the memory mapped UPDATE register in the L1 cache. The VCI CMD field contains the VCI_WRITE code. As the memory cache controller can handle several simultaneous update/invalidate transactions, the VCI PKTID field contains the transaction index. The VCI PLEN field contains the value 4*N, where N is the actual number of modified words in the cache line. The line index (34 bits) is transported in the VCI WDATA and VCI BE fields (the two LSB bits), of the first flit. The first modified word index (3 bits) is transported in the WDATA field of the second flit, and the N modified words in the WDATA and BE fields of the N following flits. For each modified word, the VCI BE field can have a different value (including the 0x0 value). The VCI response packet contains one single flit. The memory cache controller counts the number of VCI responses to detect the completion of the MULTICAST_UPDATE transaction.

• A MULTI_INVAL transaction is a multi-cast transaction, that is composed of several VCI transactions. When a memory cache makes a cache line replacement (following a MISS in the memory cache), and the victim line has a number of copies smaller than the DHCCP threshold, it sends as many VCI transactions as the number of registered copies. Both the VCI command packet and the VCI response packet contain only one flit. The VCI ADDRESS field contains the address of the memory mapped INVAL register in the L1 cache. The VCI CMD field contains the VCI_WRITE code. As the memory cache controller can handle several update/invalidate transactions simultaneously, the VCI PKTID field contains the transaction index.The VCI WDATA and VCI BE (the two LSB bits) fields contain the 34 bits line index. The memory cache controller counts the number of VCI responses to detect the completion of the MULTI_INVAL transaction.

• A BROADCAST_INVAL transaction is a broadcast transaction. This transaction is initiated when a memory cache controller replaces a line, or receives a WRITE request to a replicated cache line, and this cache line has a number of copies larger than the DHCCP threshold. The VCI command packet contains one single flit. This packet is replicated and dynamically broadcasted by the network itself. The VCI CMD field contains the VCI_WRITE code. The VCI ADDRESS field contains the global broadcast address 0x0000000003 (only the two LSB bits are set). The VCI WDATA and the VCI BE (the two LSB bits) field contain the line index. This VCI command is broadcasted to all L1 caches in the system, but only L1 caches that have a copy send a VCI response packet. All VCI response packets are independently returned to the memory cache initiator, that counts the number of VCI responses to detect the completion of the BROADCAST_INVAL transaction. If a L1 cache contains two copies of a cache line (i.e. the line is replicated in both the DATA cache, and the INSTRUCTION cache), it must send two VCI responses.

The following table defines the coherence command encoding (4 LSB bits in the VCI ADDRESS field)

COMMAND TYPE
Invalidate Data	0000
Invalidate Instruction	0100
Update Data	1000
Update Instruction	1100

• A CLEANUP transaction is initiated by a L1 cache controller to a memory cache controller, to signal that a cache line copy has been removed from an instruction or data cache. Both the VCI command packet and the VCI response packet contain one single flit. For a CLEANUP transaction, the VCI ADDRESS field must contain the removed cache line address. The VCI TRDID fiels contains the value 0 for a data cache cleanup, and contains the value 1 for an instruction cache cleanup.

2.3 External transactions

These transactions are initiated by the memory caches, to fetch or save a complete cache line in case of MISS in the memory cache. The general policy between the memory caches and the external memory is WRITE_BACK : The external memory is only updated in case of line replacement. The target is always the external RAM controller.

All the external transactions use a separated external network, implementing a separated address space. The memory cache and the external RAM controller ports used to access the external network respect a simplified version of the VCI advanced format : the VCI fields PLEN, PKTID, CONST, CONTIG and BE are not used. The VCI ADDRESS field contains 30 bits (a 64 bytes cache line index). The VCI WDATA & RDATA fields contain 64 bits, in order to improve the bandwidth. The VCI SRCID field contains the memory cache index (cluster index). As the memory cache controller can process several external transactions simultaneously, the VCI TRDID field contains the transaction index.

• For a GET transaction, the VCI command packet contains one single flit. The VCI CMD field contains the READ value. The VCI response packet contains 8 flits (corresponding to the 64 bytes of a cache line).

• For a PUT transaction, the VCI command packet contains 8 flits. The VCI CMD field contains the WRITE value. The VCI response packet contains 1 flit.