JP4868246B2 - Multiprocessor and memory replacement method - Google Patents

Description

  The present invention relates to a dynamic memory replacement method in a multiprocessor system employing a sparse directory method.

  In the dynamic replacement of the memory in the multiprocessor system adopting the sparse directory method, data is copied from the memory before the replacement to the memory after the replacement. The full directory method manages the status and the cached processor for all memories, whereas the sparse directory method manages only a part of the memory. A related memory dynamic replacement method in a multiprocessor system employing a sparse directory method will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a multiprocessor system including a plurality of cells. A case where dynamic replacement is performed from the memory 111 provided in the cell 100 to the memory 214 provided in the cell 200 will be described with reference to FIG. The cells 100 and 200 include one or more processors, one or more memories, and a cell controller.

  In the related method, the sparse directory of the copy destination cell controller 220 is updated after data is copied between the memories. For this reason, when accessing a line that has never been accessed after copying data between memories, it is necessary to broadcast a snoop to all the processors. Further, since the status of all memories cannot be managed in the sparse directory, it is necessary to read the entire copy destination memory 214 once after copying the data in the memory and update the sparse directory in order to guarantee coherency.

  The memory data copy and the operation after the end in the multiprocessor system adopting the sparse directory method will be described with reference to the flowchart of FIG.

  First, the memory data is copied from the memory 111 of the copy source processor 101 to the memory 211 of the copy destination processor 201 (S1001). Specifically, the following operation is performed. The processor 102 receives an instruction from the system FW, reads all lines of the memory 111 connected to the copy source processor 101 for each cache line, and writes to the memory of the copy source processor 101. Each time the copy source processor 101 receives a write request, the copy source processor 101 writes to the memory 111 and transfers the write to the memory 214 connected to the copy destination processor 204 (performs a write request). In this way, data copy of the memory is performed.

  After the reading and writing of all the lines of the memory 111 are completed, the system FW changes the register indicating the correspondence between the address and the memory in the system from the copy source memory 111 to the copy destination memory 214, and the copy source processor 101 and the memory 111 is disconnected from the system (S1002). Next, according to an instruction from the system FW, an arbitrary processor in the system reads the memory 214 connected to the copy destination processor 204 for each cache line (S1003). The copy destination processor 204 transmits a snoop to all the processors 201, 202, 203 and the cell controller 206 to be connected (S1004). The cell controller 220 that has received the snoop determines whether the sparse directory 221 holds the status information of the line that has received the snoop (S1005).

  When the line status information is held in the sparse directory 221 (Yes in S1006), the cell controller 220 transmits a snoop to a specific processor that caches the line based on the managed status information (S1007). ), A snoop response is received from the processor (S1008). On the other hand, when the line status information is not held in the sparse directory 221 (No in S1006), the cell controller 220 broadcasts the snoop to all the processors outside the cell 200 (S1009), and all the snoop responses are transmitted. Receive (S1010).

  Next, the cell controller 220 updates the sparse directory 221 based on the snoop response (S1011), and transmits a snoop response to the copy destination processor 204 (S1012). The copy destination processor 204 receives all the snoop responses and transmits data to the processor that transmitted the request (S1013).

  The arbitrary processor confirms whether or not reading of all the lines in the copy destination memory 214 has been completed, and if completed (Yes in S1014), shifts to a normal operation (S1015), and there remains an unread line. In the case (No in S1014), the read point is aligned with the unread line in the cache line of the copy destination memory 214, the read is performed (S1016), and the processing from Step S1004 is repeated. In this way, a series of operations are performed for the total number of cache lines in the memory of the copy destination processor 204.

As a result, the cache information of the processors outside the cell 200 is managed by the sparse directory 221 in the cell controller 220, and inter-processor cache coherency is guaranteed. Patent Document 1 discloses a system for memory migration in a distributed memory multiprocessor system. In this technology, every time one line of memory is copied, the access destination line is changed from the old line to the new line, and the access of the changed line is transferred from the old line to the new line.
JP 2004-054931 A

  However, in this related method, in order to guarantee coherency between processor caches, all cache lines in the copy destination memory are read, and snoop broadcast is performed to all processors included outside the copy destination cell (cell 200 in the above example). There was a need. For this reason, there is a problem that the processing time of the dynamic replacement of the memory is long, and the throughput of other requests is reduced during the read after the data copy of the memory, and the performance is deteriorated. In recent years, the memory capacity per system has been increasing, and it has been necessary to solve this problem.

  The present invention has been made in view of such circumstances, and in a multiprocessor system employing a sparse directory method, the memory operation is improved by updating the copy destination sparse directory using the copy source sparse directory. The purpose is to speed up the process of automatic replacement.

  One aspect of the multiprocessor according to the present invention is a multiprocessor system including a plurality of cells each having one or more processors having a cache memory and one or more memories, wherein each cell is a cache memory in another cell. When copying data between the sparse directory that stores the status information related to the memory data in the own cell and the memory in which the copy source cell and copy destination cell are different, the data is copied from the copy source memory to the copy destination memory. A cell controller that performs copying and updates the sparse directory of the copy destination cell based on status information held in the sparse directory in the copy source cell.

  Another aspect of the memory replacement method according to the present invention is a memory replacement method for a multiprocessor system including a plurality of cells each having one or more processors having a cache memory and one or more memories. Status information regarding the data in the memory in its own cell held in the cache memory of the cell, and copying the data from the copy source memory to the copy destination memory between memories with different cells. The access destination is changed to the copy destination memory, and the status information held in the sparse directory of the copy destination cell is updated based on the status information held in the sparse directory in the copy source cell.

  According to the present invention, in a multiprocessor system employing a sparse directory system, it is possible to speed up the dynamic replacement process of a memory by updating the copy destination sparse directory using the copy source sparse directory. It becomes.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In the drawings, components having the same configuration or function and corresponding parts are denoted by the same reference numerals, and description thereof is omitted.

(Embodiment 1)
In the present embodiment, an aspect will be described in which the cache state of the copy source memory is reflected in the copy source sparse directory, and the information of the copy source sparse directory is reflected in the copy destination sparse directory after data is copied between the memories.

  FIG. 1 is a block diagram illustrating a configuration example of a cell included in a multiprocessor system according to an embodiment of the present invention. The cell 100 is one unit of the multiprocessor system, and includes processors 101, 102, 103, and 104, memories 111, 112, 113, and 114, and a cell controller 120.

  The cache (cache memory) 131, 132, 133, 134 employs a general cache protocol such as MESI (Modified Exclusive Shared Invalid) that temporarily stores memory data in the processors 101, 102, 103, 104. Cache memory.

  The processors 101, 102, 103, and 104 are connected to the memories 111, 112, 113, and 114, respectively, and manage data in each memory. The processors 101 to 104 and the cell controller 120 are connected one-to-one by the interfaces 151 to 160.

  The memories 111 to 114 are accessed from each processor in units of cache lines. The bit width of one cache line is, for example, 64 bytes, and the position of the cache line is determined by an address (address 303 in FIG. 3 described later) specified at the time of memory access.

  The cell controller 120 controls access between the memories 111 to 114 in the cell 100 and the outside of the cell 100. When the cell controller 120 receives an access from the outside of the cell 100, the cell controller 120 holds information on whether or not the memories 111 to 114 are cached in an arbitrary processor, and information on the processor number if cached. A sparse directory 121 is provided.

The sparse directory 121 manages all the cache statuses held by the processors of other cells with respect to the memories 111 to 114 in the cell. Here, the sparse directory 121 has fewer entries than all the cache lines of the memories 111 to 114 in the cell 100, and when the number of manageable entries is exceeded, a mechanism for sweeping data from the processors in other cells. Have
A system FW (Firmware) 122 is software that is provided in the cell controller 120 and performs basic control of HW (hardware). For example, data copying between memories this time can be said to be one that can be controlled by the system FW. .

  Note that the number of processors and memories shown in FIG. 1 is an example, and it is sufficient that one cell includes one or more processors and memories.

  FIG. 2 shows connections between the cells shown in FIG. FIG. 2 is a diagram illustrating an example of a multiprocessor system including a plurality of cells. FIG. 2 shows an example in which cells 200, 500, and 550 are provided in addition to the cell 100. The cells 200, 500, and 550 have the same configuration as the cell 100, but are partially omitted in FIG. The cell controllers 120, 220, 520, and 570 are connected on a one-to-one basis through interfaces 251 to 256, and transfer access between the cells 100, 200, 500, and 550. The cell 200 includes processors 201, 202, 203, 204, a memory 214, a cell controller 220, and a sparse directory 221 installed in the cell controller 220. The cell 500 includes processors 501, 502, 503, and 504 and a cell controller 520, and the cell 550 includes processors 551, 552, 553, and 554, and a cell controller 570. The cells 200, 500, and 550 have components corresponding to the caches 105 to 108, the memories 111 to 114, the sparse directory 121, and the system FW 122 shown in FIG.

  FIG. 3 is a diagram showing a format of a packet used in communication between cells in the embodiment of the present invention. The packet 300 includes a flag 301, a request type 302, an address 303, a cache status 304, a transmission destination processor number 305, a reply destination processor number 306, and data 307.

  The flag 301 is a flag indicating that the directory is being updated, and is used by the copy source cell controller or the copy destination cell controller. The flag 301 indicates that the packet is a normal packet when “0”, and indicates that the packet is a dedicated packet for directory update when “1”. A cell controller that is neither the copy source nor the copy destination carries the value of the flag 301. The flag 301 is set to “1” by the request / snoop generation circuit 410 of the copy source cell controller, which will be described later with reference to FIG. 4, and is checked by the first arbitration circuit 402 and the write data generation circuit 406 of the copy destination cell controller. The

  The request type 302 is a field indicating the type of request requested by the packet, and stores values corresponding to read, read reply, write, snoop, and snoop response. Read reads data from the memory and requests to send data. The read reply is a reply to the read, and transmits the read data to the processor that requested the read. The snoop requests the cached processor to change to the status indicated by the cache status 304 based on the status information of the read address. The snoop response is a reply to the snoop, and the processor that has received the snoop transmits the snoop response after changing the status of the cache line indicated by the address to the status requested by the snoop.

  An address 303 is a field indicating a memory address, and indicates which cache line of which memory is accessed. The address 303 is also used to determine a sparse directory entry.

  The cache status 304 is a field indicating the MESI status of the line specified by the address 303. When the packet request type 302 is read reply, the cache status 304 indicates the status of the data 307. When the request type 302 is snoop response or write, the cache status 304 indicates the status of the cache line specified by the address 303 held by the processor that sent the request. When the request type 302 is read, the cache status 304 indicates a status requested by the transmission source processor to the line specified by the address 303. When the request type 302 is snoop, the status of changing the line is indicated to the destination processor.

  The transmission destination processor number 305 indicates the processor number to which the request (snoop, read, write) as a cause is transmitted (received). That is, the transmission destination processor number 305 indicates the processor number that caches the line specified by the address 303 when the request type 302 is snoop and snoop response, and when read, read reply, or write, the memory to be accessed Indicates the connected processor number.

  The transmission destination processor number 305 indicates a processor number that transmits a request (snoop, read, write) as a cause. That is, when the request type 302 is snoop and snoop response, the reply destination processor number 306 indicates the processor number to which the memory indicated by the address is connected. When the request type 302 is read, read reply, or write, the processor number that transmitted the request is indicated. Show. The value of the reply destination processor number 306 in the snoop oscillated from the copy source cell controller 120 will be described later.

  Data 307 is a part for storing data, and is used for read reply, write, and snoop response.

  FIG. 4 shows a configuration example of the cell controller according to the embodiment of the present invention. In FIG. 4, the cell controller 120 in the cell 100 is shown as an example, but cell controllers provided in other cells also have the same configuration and function. In addition to the sparse directory 121 and the system FW 122 shown in FIG. 1, the cell controller 120 includes a status flag 401, a first arbitration circuit 402, a request storage buffer 403, a snoop response storage buffer 404, an address generation circuit 405, a write data generation circuit ( Sparse directory write data generation circuit) 406, write command generation circuit 407, read pointer 408, selectors 409 and 413, request / snoop generation circuit (generation unit) 410, synthesis circuit 411, and second arbitration circuit 412.

  The status flag 401 indicates whether the cell controller 120 is in a normal state, is operating as a copy source cell controller after copying data in the memory, or operates as a copy destination cell controller after copying data in the memory. It is a flag indicating whether or not the current state is in progress. The status flag 401 is set by the system FW 122. The number of bits of the status flag 401 is, for example, 2 bits, [00] indicates that it operates in a normal state, [01] indicates that it operates as a copy source cell controller, and [10] indicates that it operates as a copy destination cell controller. Indicates that it operates, and [11] is not used. The 2-bit information is output to the first arbitration circuit 402, the write data generation circuit 406, and the request / snoop generation circuit 410. Further, it is output as a selector condition for the address generation circuit 405 and the read pointer 408.

  The first arbitration circuit 402 arbitrates accesses from the processors 101 to 104 in the cell 100 and accesses from the cell controllers 220, 520, and 570 outside the cell 100, and outputs them to the request storage buffer 403 or the snoop response storage buffer. To do. The first arbitration circuit 402 receives the packet 300 and outputs it to the request storage buffer 403 when the status flag 401 is [00] or [01]. When the status flag 401 is “10”, the first arbitration circuit 402 checks the flag 301 indicating that the sparse directory 121 is being updated in the packet. If the flag 301 is “1”, the first arbitration circuit 402 snoops the received packet. When the flag 301 is not “1”, the request is stored in the request storage buffer 403.

  The request storage buffer 403 holds all requests currently being executed by the cell controller 120. When a snoop is transmitted from the request / snoop generation circuit 410 due to the held request, snoop information is held in the same entry as the request, and the snoop response is received.

  When reading the entry of the sparse directory 121, the cell controller 120 transmits the address 303 of the held request to the address generation circuit 405 and specifies the entry. When updating the entry of the sparse directory 121, the address 303 of the held request is transmitted to the address generation circuit 405, and the status of the memory line corresponding to the address (cache status 304 and cache processor number) is transmitted to the write data generation circuit 406. Send. When the memory connected to the processor needs to be updated, the request type 302 and the data 307 are transmitted to the write command generation circuit 407. When the memory update is not necessary, the request type 302 in the packet is output to the request / snoop generation circuit 410.

  The snoop response storage buffer 404 is used when it is a copy destination cell controller. The type of request for a packet received by the snoop response storage buffer 404 is only a snoop response. In the snoop response storage buffer 404, when the status flag 401 is [10], that is, when the cell controller 120 becomes a copy destination cell controller after data copy of the memory, and when the flag 301 of the packet 300 is “1”. That is, when the sparse directory is being updated, the packet is received from the first arbitration circuit 402. For this reason, the snoop response for updating the sparse directory stored in the snoop response storage buffer 404 is also referred to as an “update snoop response”. The cell controller 120 updates the sparse directory 121 based on the received snoop response information. That is, the address 303 is transmitted to the address generation circuit 405 in order to determine the entry of the sparse directory, and the memory line corresponding to the address is transmitted to the write data generation circuit 406 in order to generate status information (cache status information) to be updated. Status information (cache status 304 and cache processor number) is transmitted. If the memory connected to the processor needs to be updated, the request type 302 and data 307 are transmitted to the write command generation circuit 407.

  The address generation circuit 405 creates a key address from the address 303 transmitted from the request storage buffer 403 or the snoop response storage buffer 404, and determines an entry of the sparse directory 121 to be accessed. The address generation circuit 405 is selected by the selector 409 when the status flags are [00] and [10], and can access the sparse directory. The method for creating the key address from the address is determined depending on the configuration of the sparse directory. The details of the method are not mentioned here. Based on the key address of the determined entry, the sparse directory 121 is accessed.

  The write data generation circuit 406 generates data to be written to the sparse directory 121 from information sent from the request storage buffer 403 or the snoop response storage buffer 404. The data to be written is a processor number that caches the line and a cache status value of the line, specifically, one of ME (Modified Exclusive), S (Shared), and I (Invalid). In this embodiment, in the packet 300, it is always checked whether the flag 301 is “1” (indicating that the directory is being updated). If it is not “1”, the status information of the sparse directory 121 is always updated. When the flag 301 is “1” and the status information is not registered in the sparse directory 121, the status information of the sparse directory is updated. When the status information is registered, the status information is updated. Absent.

  The write command generation circuit 407 transmits data corresponding to a read request from a processor in another cell 200, 500, or 550 when writing data to the memories 111 to 114 connected to the processors 101 to 104 in the cell 100. In this case, a read reply command and a write command are generated based on the request type 302 and the data 307 included in the packet 300 output from the request storage buffer 403 or the snoop response storage buffer 404, and output to the synthesis circuit 411.

  The read pointer 408 includes a register indicating a threshold value of the number of entries in the sparse directory 121 and a read counter indicating an entry when the sparse directory 121 is read as a copy source cell. The initial value of the read counter is “0”. The read pointer 408 is selected by the selector 409 when the state flag 401 of the cell controller 120 is asserted to [01] (that is, the copy source cell controller after the data copy of the memory), and based on the read counter Access an entry in the sparse directory 121. When the read pointer 408 receives the strobe signal from the request / snoop generation circuit 410, the read pointer 408 increments the request counter by +1 ("1" increment), and starts reading the next entry in the sparse directory. The cell controller 120 has a function of accessing the entries of the sparse directory until the threshold value of the number of entries (register value) is exceeded.

  The selector 409 changes the selection according to the value of the status flag 401. When the status flag 401 is [00] or [10], the address generation circuit 405 is selected, and when the status flag 401 is [01], the read pointer 408 is selected. However, even when the status flag 401 is [01], since there are other processors in the cell 100 that are not involved in data copying between memories, the request is stored in the request storage buffer 403 (others in the cell 100). The address generation circuit 405 is preferentially selected if the request is received from the other processor in the cell 100 or to the memory connected to another processor in the cell 100).

  The sparse directory 121 manages status information for each line regarding the memories 111 to 114 in the cell 100. The status information holds information including whether the line is cached in a processor in another cell and information including the cached processor number when the line is cached in a processor in another cell. Yes. Further, all the lines of the memories 111 to 114 in the cell cached by the processor outside the cell 100 are managed. As a method of address mapping between the memory and the processor cache, for example, there is a set associative method, but in the present invention, any method may be used as long as the correspondence between the address and the entry can be taken. The sparse directory 121 has a function of writing status information generated by the write data generation circuit 406 to an entry corresponding to the key address specified by the address generation circuit 405. Further, it has a function of reading an entry corresponding to the key address instructed by the address generation circuit 405 and outputting status information of the read address to the request / snoop generation circuit 410 and the synthesis circuit 411. Further, it has a function of reading an entry instructed by the read counter of the read pointer 408 and outputting the read address and status information of the address to the request / snoop generation circuit 410.

  The request / snoop generation circuit 410 verifies the necessity of snoop based on the address received from the sparse directory 121, the status information of the address, and the request type 302 received from the request storage buffer 403 or the snoop response storage buffer 404. . The request / snoop generation circuit 410 generates a snoop request and transmits it to the second arbitration circuit 412 when it is determined from the status information that the type 302 of the request is snoop or read and the processor of another cell caches. To do. At the same time, the request storage buffer 403 is notified of snoop information. When the request type 302 is a snoop response, the status information (MESI) of the line corresponding to the address received from the sparse directory 121 is attached and transmitted to the processor.

  Further, the request / snoop generation circuit 410, when the cell controller 120 is a copy destination cell controller after the data copy of the memory, specifically, when the status flag 401 is asserted to [10], the sparse directory 121. When a request or snoop that does not hold status information corresponding to the address of the request is received, a snoop broadcast mechanism is provided to all the processors outside the cell 100 and is output to the second arbitration circuit 412.

  Further, when the cell controller 120 is a copy source cell controller after data copy of the memory, specifically, when the status flag 401 is asserted to [01], the request / snoop generation circuit 410 has the sparse directory 121. When there is a snoop instruction from, a mechanism for generating a snoop in which a reply destination processor number 306 of the snoop packet is set in the copy destination cell controller and a flag 301 indicating that the directory is being updated is set to “1”. At the same time when the generated snoop is output to the second arbitration circuit 412, a strobe signal is transmitted to the read pointer 408 in order to increment the read pointer held by the read pointer 408 by +1 ("1" increment).

  The synthesizing circuit 411 synthesizes the read reply or write generated by the write command generation circuit 407 and the status information (MESI) of the line corresponding to the address received from the sparse directory 121, and outputs it to the second arbitration circuit 412. .

The second arbitration circuit 412 arbitrates transmission of the snoop, read, and snoop response output from the request / snoop generation circuit 410, and transmission of the read reply and write output from the synthesis circuit 411, and the processor 102 of the cell 100, 103 and 104 (excluding processors corresponding to the copy source memory) and cell controllers 220, 520, and 550 outside the cell 100.
The selector 413 selects the output of the request storage buffer 403 and the snoop response storage buffer 404. The selector 413 receives the value of the status flag 401 and selects the output of the snoop response storage buffer 404 with priority when the status flag 401 indicates [10], that is, when it is a copy destination cell. Good.

  Subsequently, the data copy operation of the memory will be described with reference to FIG. 1, FIG. 2, and FIG. FIG. 5 is a flowchart illustrating an operation example of data copying between memories according to the present embodiment. Here, an operation when data in the memory 111 of the cell 100 is copied to the memory 214 of the cell 200 will be described as an example.

  In order to manage the cache status of the memory 111 connected to the processor 101 including the caches of the processors 102 to 104 in the copy source cell 100 with the sparse directory 121 of the cell controller 120 before copying data between the memories. In the cell 100, the routes 155, 156, and 157 with the processors 102, 103, and 104 connected to the processor 101 are changed so as to pass through the cell controller 120. Specifically, 155 is changed to 152 and 151, 156 is changed to 153 and 151, and 157 is changed to 154 and 151 (S601). As a result, the sparse directory 121, which normally has only the cache information of the processors of other cells, reflects the cache status of the processors 102, 103, and 104 in the cell regarding the memory 111.

  Next, in response to an instruction from the system FW 122, the processor 102 (which may be any processor inside or outside the cell 100) reads all lines of the memory 111 connected to the copy source processor 101 for each cache line, and Write to the memory 111 (write request). Each time the copy source processor 101 receives a write request, the copy source processor 101 writes to the memory 111 and transfers the write to the memory 214 connected to the copy destination processor 204 (performs a write request). Copying is performed for all lines line by line (S602).

  The system FW 122 changes the access destination and sets the status flag 401 (S603). Specifically, the system FW 122 changes the processor connected to the memory to be accessed from the copy source processor 101 to the copy destination processor 204, and the copy source processor 101 is disconnected from the system. Regarding the change of the processor to be accessed and the method of disconnecting the copy source processor, a general method is adopted in the present embodiment, and thus detailed description thereof is omitted. After the copy is completed, the system FW 122 sets the status flag 401 in the cell controller to [01] for the copy source cell controller 120 and [10] for the copy destination cell controller 220.

  The copy source cell controller 120 checks the status information for each entry held in the sparse directory 121 in the cell controller 120 (S604). Specifically, the status information of the entry indicated by the read counter held by the read pointer 408 is confirmed. Based on the status information, the request / snoop generation circuit 410 of the copy source cell controller 120 determines a processor that caches the data in the memory 111 based on the status information of the sparse directory, and returns a reply processor number 306 of the snoop packet. Is changed to the copy destination cell controller 220, and a snoop in which the flag 301 indicating that the directory is being updated is set to “1” is transmitted to a specific processor that caches (605). Detailed internal operation of the copy source cell controller 120 will be described later.

  The specific processor that received the snoop checks whether there is a line indicated by the address 303 of the snoop packet in the cache in its own processor, and if the data is the latest, sends a snoop response with write data, otherwise Transmits a snoop response with no data to the copy destination cell controller 220 while holding the flag 301 indicating that the directory is being updated.

  The copy destination cell controller 220 receives a snoop response in which the flag 301 indicating that the directory is being updated is “1” from the specific processor (S606), and stores the status information of the line indicated by the address 303 of the snoop response packet in the sparse directory 221. It is determined whether it is held (S607). If it is held (Yes in S607), the snoop response is discarded without updating the sparse directory 221 (S608). On the other hand, when the status information is not held (No in S607), the information of the sparse directory 221 is updated (609). Detailed internal operation of the copy destination cell controller 220 will be described later.

  The copy source cell controller 120 increments the read counter of the read pointer 408 by “1” (S610). The copy source cell controller 120 determines whether the read counter value of the read pointer 408 exceeds the number of entries in the sparse directory 121 (S611). When the number of entries (the value of the register indicating the threshold value of the sparse directory entry of the read pointer 408) is exceeded (Yes in S611), the cell controller 120 shifts the status flag 401 to the normal operation (S612). If the number is less than the number of entries (No in S611), the cell controller 120 confirms the status information of the cache line of the sparse directory 121 indicated by the read counter of the read pointer 408 (S613), and repeats the processing from step S605. A series of operations is performed for the number of entries in the sparse directory 121 of the copy source cell controller 120.

  In this way, the copy source cell controller 120 holds the status information of all the processors accessing the memory 111 connected to the processor 101 in the sparse directory 121 by the path change, and based on the status information of the sparse directory 121. Send a snoop. The cell controller 220 of the copy destination cell updates the sparse directory 221 of the copy destination cell controller 220 based on the snoop response to the snoop transmitted from the copy source cell controller 120. Thereby, the coherency between processor caches can be ensured.

  Here, detailed internal operations of the copy source cell controller and the copy destination cell controller will be described with reference to FIGS.

  The copy source cell controller 120 selects the read pointer 408 when the status flag 401 is set to [01]. The request / snoop generation circuit 410 reads the entry of the sparse directory 121 indicated by the read counter of the read pointer 408 by HW (Hardware). Based on the status information of the read entry, the request / snoop generation circuit 410 sets the packet transmission destination processor number 305 in the processor that caches the destination so that the reply destination is the copy destination cell controller 220. “1” indicating that the directory is being updated in the packet is set in the flag 301. The generated packet 300 is transmitted to the second arbitration circuit 412.

  The second arbitration circuit 412 performs arbitration with other snoops and requests, and if selected by arbitration, transmits the snoop to the cached processor. The request / snoop generation circuit 410 transmits to the second arbitration circuit 412 and simultaneously transmits a strobe signal to increment the read counter of the read pointer 408 by “1”. The read pointer 408 that has received the strobe signal The read counter is incremented by “1” (S610), the value of the read pointer 408 is compared with the number of entries in the sparse directory 121 (S611), and if the value of the read pointer 408 exceeds (Yes in S611), the entry is read. And the status flag is changed to [00] (S612). If not exceeded (No in S612), the sparse directory 121 is accessed (S613).

  When the status flag 401 is set to [10], the copy destination cell controller 220 checks whether the flag 301 indicating that the directory in the packet is being updated is “1”. The packet set to 1 ″ is stored in the snoop response storage buffer 404. The snoop response storage buffer 404 outputs the address 303 to the address generation circuit 405 and the status information of the line of the memory 214 corresponding to the address (the cache status 304 and the processor that performs the cache) to the write data generation circuit 406 for the stored packet. . The address generation circuit 405 generates a key address for specifying the entry number of the sparse directory 121 from the address 303. The write data generation circuit 406 generates data for updating the sparse directory 221 from the status information.

  When the cell controller 220 is registered in the entry as a result of subtracting (referring to) the sparse directory 221 with the key address generated by the address generation circuit 405 (Yes in S607), the data generated by the write data generation circuit 406 is displayed. Discard. If it is not registered in the entry (No in S607), the sparse directory 221 is updated. When the system FW 122 confirms that the status flag of the copy source cell controller has been set to [00], the status flag of the copy destination cell controller is also set to [00] and returns to the normal operation.

  Next, when the cell controller 220 is operating as the copy destination cell controller, specifically when the status flag 401 is asserted to [10], the memory 214 connected to the copy destination processor 204 is accessed. The operation when a snoop is received from the copy destination processor 204 will be described with reference to FIG. FIG. 6 is a flowchart showing an operation example when the cell controller operates as a copy destination cell controller.

  The copy destination cell controller 220 receives a request to the memory 214 connected to the copy destination processor 204 from outside the cell 200 (S701). Alternatively, the copy destination cell controller 220 receives a snoop from the copy destination processor 204 (702). When either step S701 or S702 occurs, the cell controller 220 confirms whether the status information of the accessed line is managed by the sparse directory 221 (S703).

  When managing (Yes in S703), the cell controller 220 transmits a snoop to a specific processor that performs caching based on the status information (S704), and then receives a response (S705). The sparse directory is updated (708). When not managed by the sparse directory 221 (No in S703), the snoop is broadcast to all the processors inside and outside the cell (S706), all snoop responses are received (S707), and the status information of the sparse directory 221 is updated. (S708). When the update of the sparse directory 221 is completed, the cell controller 220 transmits a request to the copy destination processor 204 when a request is received in step S701, and performs a snoop response when a snoop is received in step S702 (S709). .

  In this operation, when status information is not managed in the sparse directory 221 as in the normal state, that is, the operation when the state flag 401 is asserted to [00], a snoop response or a read request is immediately addressed. Unlike being able to transmit to the processor to which the determined memory is connected, a snoop broadcast (S706) is required to ensure inter-processor cache coherency.

  Next, operations from a request from a processor in the cell 100 and operations from a request from a processor outside the cell 100 will be described as normal operations. FIG. 7 is a flowchart showing an operation example when the cell controller of the present embodiment receives a request from the processor in the cell in the normal state. FIG. 8 is a flowchart showing an operation example when the cell controller of this embodiment receives a request from a processor outside the cell in a normal state.

  With reference to FIG. 7, a case where a read request is issued from the processor 102 in the cell 100 to the memory 111 will be described as an example. The processor 102 issues a request to the memory 111 connected to the processor 101 (S801). The processor 101 that has received the request broadcasts a snoop to the processors in the cell 100 and the cell controller 120 (S802). The cell controller 120 that has received the snoop confirms whether the status information of the sparse directory 121 is managed (S804).

  When status information is held in the sparse directory 121 (Yes in S804), a snoop is transmitted to the processor that caches the line based on the managed status information (S805), and the snoop is sent from the processor. A response is received (S806). Based on the received information, the cell controller 120 rewrites the sparse directory 221 (S807), and sends a snoop response to the processor 101 that has requested the request (S808). On the other hand, when the status information is not held in the sparse directory 121 (No in S804), the cell controller 120 immediately returns a snoop response to the processor 101 (S808).

  After receiving all the snoop responses from the cell controller 120, the processor 101 transmits data to the processor 102 that issued the read request (809). In this way, the operation from the request of the processor in the series of cells is completed.

  Next, a case where a read request is issued from the processor 201 outside the cell 100 to the memory 111 will be described as an example with reference to FIG. The processor 201 issues a request to the memory 111 connected to the processor 101 to the cell controller 120 via the cell controller 220 (S901). The cell controller 120 that has received the request checks the status information of the sparse directory 121 (S902). It is checked whether status information is managed in the sparse directory 121 (S903).

  When status information is held in the sparse directory 121 (Yes in S903), a snoop is transmitted to the processor that caches the line based on the managed information (S904), and the snoop response is sent from the processor. Is received (S905). The cell controller 120 rewrites the sparse directory 121 based on the received information (S906), and transmits the request to the processor 101 that issued the request (S907). On the other hand, when the status information is not held in the sparse directory 121 (No in S903), the request is immediately transmitted to the processor 101 (S907).

  The processor 101 that has received the request broadcasts a snoop to the processors 102 to 104 in the cell 100 (S908). After receiving all the snoop responses from the processors 102 to 104, the processor 101 transmits data to the cell controller 120 (S909). The cell controller 120 transmits data to the processor 201 (S910). In this manner, the operation from the request of the processor of another series of cells is completed.

  In the above description, the copy source cell is assumed to be the cell 100 and the copy destination cell is assumed to be the cell 200. However, it goes without saying that the data copy between other memories functions similarly.

(Other embodiments)
In the above embodiment, the case where a cell has a plurality of processors has been described. However, the present invention can be applied even when a combination of one processor and a memory is provided in a cell. In such a case, in step S601 in FIG. 5, the processor in the cell receives an instruction from the system FW and requests to read to the memory 111, but the system FW instructs the processor outside the cell to read. Also good. Further, since there is no information about the copy source memory held in the cache memory of the processor in the copy source cell, the process of collecting status information in the sparse directory in the copy source cell is not necessary.

  As described above, according to the preferred embodiment of the present invention, before the memory data is copied, the interface between the local processors in the cell is disconnected, and the copy source memory is read. Dynamic copy processing of memory by reflecting the status information of the local processor in the copy source sparse directory and updating the copy destination sparse directory using the copy source sparse directory information after copying the memory data. Can be speeded up. Specifically, the cell controller (for example, the cell controller 120 in FIG. 2) to which the copy source processor is connected reads the sparse directory of the cell controller (for example, the sparse directory 121 in FIG. 2) for each entry, and snoops. Is provided in the request / snoop generation circuit (for example, the request / snoop generation circuit 410 in FIG. 4).

  Further, a snoop for a snoop that is not transmitted to the cell controller (for example, the cell controller 220 in FIG. 2) to which the copy destination processor (for example, the processor 204 in FIG. 2) is connected (sent by the copy source cell controller). A mechanism for transmitting the response to the snoop response storage buffer (for example, the snoop response storage buffer 404 in FIG. 4) is provided in the first arbitration circuit (for example, the first arbitration circuit 402 in FIG. 4). Based on the status information of the sparse directory held by the copy source cell controller, data is swept from the caches of all processors, and the copy destination sparse directory is updated to ensure coherency and a part of the memory line is Since only snoops for the number of entries of the sparse directory to be managed (for example, the sparse directory 121 in FIG. 2) are issued, the sparse directory (for example, the sparse directory 221 in FIG. 2) can be updated at high speed after the data copy of the memory. Furthermore, since the number of snoops per line can be suppressed to only snoops to the processor that performs caching, an increase in traffic due to snoops can be suppressed. In this way, the number of issued snoop broadcasts is suppressed, and when dynamic memory replacement is performed, processing is performed without lowering the throughput of other requests than before.

Moreover, according to a preferred embodiment of the present invention, the effects described below can be obtained.
The first effect is that in order to guarantee interprocessor cache coherency, conventionally, it is necessary to read all the number of cache lines in the memory after the data copy of the memory is completed. However, in the present invention, all entries of the sparse directory that manages a part of the cache lines of the memory may be read. Since the number of entries is much smaller than the number of lines in the memory, the time taken to guarantee cache coherency can be reduced.

  The second effect is that, conventionally, a snoop is transmitted to all processors for one cache line of the memory when the memory is read after the data copy of the memory is completed. However, in the present invention, by using the sparse directory of the copy source cell controller, it is only necessary to send a snoop to a specific processor for one cache line, so traffic can be reduced and performance degradation can be suppressed. Can do.

  In addition, this invention is not limited to embodiment shown above. Within the scope of the present invention, it is possible to change, add, or convert each element of the above-described embodiment to a content that can be easily considered by those skilled in the art.

It is a block diagram which shows the structural example of the cell with which the multiprocessor system which concerns on embodiment of this invention is provided. It is a figure which shows an example of the multiprocessor system comprised from a some cell. It is a figure which shows the format of the packet used by the communication between cells in this embodiment. It is a block diagram which shows the structural example in the cell controller which concerns on this embodiment. It is a flowchart which shows the operation example of the data copy between the memory of this embodiment. It is a flowchart which shows the operation example in case the cell controller of this embodiment is operate | moving as a copy destination cell controller. It is a flowchart which shows the operation example when the cell controller of this embodiment receives a request from the processor in a cell in a normal state. It is a flowchart which shows the operation example when the cell controller of this embodiment receives a request from the processor outside a cell in a normal state. It is a flowchart which shows the operation example of the data copy between the memories of related technology.

Explanation of symbols

100, 200, 500, 550 Cell 101, 102, 103, 104 Processor 111, 112, 113, 114 Memory 120 Cell controller 121 Sparse directory 122 System FW
131, 132, 133, 134 Cache 300 Packet 301 Flag 302 Request type 303 Address 304 Cache status 305 Transmission destination processor number 306 Return destination processor number 307 Data 401 Status flag 402 First arbitration circuit 403 Request storage buffer 404 Snoop response storage buffer 405 Address generation circuit 406 Write data generation circuit 407 Write command generation circuit 408 Read pointer 409, 413 Selector 410 Request / snoop generation circuit 411 Synthesis circuit 412 Second arbitration circuit

Claims (9)

A multiprocessor system comprising a plurality of cells having one or more processors having a cache memory and one or more memories,
Each cell is
A sparse directory that holds status information about the data in the memory in its own cell recorded in the cache memory in another cell;
When copying data between memories in which the copy source cell and the copy destination cell are different, the data is copied from the copy source memory to the copy destination memory, and the copy is performed based on the status information held in the sparse directory in the copy source cell. And a cell controller that updates a sparse directory of the destination cell.   2. The multiprocessor system according to claim 1, wherein the cell controller issues a snoop whose return destination is a cell controller of the copy destination cell based on status information held in a sparse directory of the copy source cell. . The memory is accessed via a connected processor;
The cell includes a plurality of processors, and when the data copy is performed, the connected processors are connected to each other so that the connected processor accesses another processor in the own cell via the cell controller,
The cell controller, when accessing the copy source memory, status information about the data of the copy source memory held in the cache memory by another processor in the own cell, and status information held by the copy source sparse directory The multiprocessor system according to claim 2, further comprising:   When the cell controller is the copy source cell, the cell controller specifies a processor that holds the data of the copy source memory in the cache memory based on status information about the copy source memory, and sets the cell controller of the copy destination cell. The multiprocessor system according to any one of claims 1 to 3, wherein a snoop as a reply destination is generated and the generated snoop is transmitted to the specified processor.   The cell controller, when it is the copy destination cell, receives a snoop response transmitted from the specified processor, and updates status information of a sparse directory based on the received snoop response. 5. The multiprocessor system according to 4.   When the cell controller is the copy destination cell, the cell controller receives a request to access the copy destination memory while updating the status information of the sparse directory, and the memory line accessed by the received request is the status information of the sparse directory. 6. The multiprocessor system according to claim 5, wherein if it is determined that the sparse directory is not included, the sparse directory is updated based on the returned snoop response.   7. The cell controller changes the access destination to the copy source memory to a copy destination memory after the data copy between the memories is completed, and updates the status information of the sparse directory. The multiprocessor system according to any one of the above. The cell controller is
A status flag indicating whether the cell is a normal state, a copy source cell, or a copy destination cell;
A request storage buffer for storing requests for access to the cell;
A snoop response storage buffer for storing a snoop response issued by the cell controller of the copy source cell as an update snoop response;
An arbitration circuit that receives the request and the update snoop response and stores the request in either the request storage buffer or the update snoop response based on a status flag;
A generation unit that generates a snoop and issues it to all processors when the status flag is a copy destination cell and the request is an access to a memory line that does not hold status information in the sparse directory; Prepared,
The sparse directory is updated based on the snoop response for update and a snoop response returned for the snoop issued by the generation unit. Multiprocessor system. A memory replacement method for a multiprocessor system comprising a plurality of cells each having one or more processors having a cache memory and one or more memories,
Holds status information about data in the memory in its own cell held in the cache memory in another cell,
Copy data from copy source memory to copy destination memory between memories with different cells,
After copying, change the access destination to the copy source memory to the copy destination memory,
A memory replacement method for updating status information held in a sparse directory in the copy destination cell based on status information held in a sparse directory in the copy source cell.

Images