JPH0628200A - Failure recovery processing method - Google Patents

Abstract

(57) [Summary] [Purpose] It is an object of the present invention to reduce the time from the detection of a failure to the execution of recovery measures in a failure recovery processing method in a failure monitoring device. [Structure] Under the control of the processing device 1, the monitoring / measurement executing means 2 scans a plurality of monitoring points 3 for displaying the presence / absence of a fault in the hardware 7 and sequentially stores the monitoring information in the storage means 4. After that, the monitoring processing means 5 in the processing device 1 sequentially reads the monitoring information, determines the measures for each monitoring point, stores the measures information in the storage means 4, and then stores the measures information in the monitoring / measures executing means 2. A failure recovery processing method in a failure monitoring apparatus for reading and executing measures, comprising: a processing apparatus 1
When a notification is received from the monitoring processing means 5 that a monitoring point displaying a failure is received, the monitoring / measurement executing means 2 monitors so that the measure information for the monitored point is stored in the storage area first read. The processing means 5 is configured to include the measure information storage area changing means 6 designated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a failure recovery processing method in a failure monitoring device. 2. Description of the Related Art In recent years, a fault monitoring apparatus that monitors the presence or absence of faults in a plurality of hardware and takes appropriate measures when a fault is found is used in many systems.

In the above fault monitoring apparatus, a supervisory control circuit for monitoring faults and taking measures scans the surveillance points displaying the presence / absence of a fault in each hardware, and the display content of each surveillance point is used as the monitoring information. When the data is sequentially written in the memory, the processing device sequentially reads this monitoring information, determines the measure for each monitoring point, writes the determined measure content in the memory as the measure information, and then the monitor control circuit reads the measure information. It is generally configured to execute.

The above-mentioned measures are mainly to disconnect a faulty device corresponding to a monitoring point where a fault is detected, or switch to a standby device (in the case of a duplex configuration). In many cases, no particular notice is given that no action is required.

In this configuration, since the order of the hardware for monitoring and taking measures by the processor and the supervisory control circuit is usually fixed, the measure information written in the memory by the processor is irrespective of whether or not there is a failure. It is read out by the supervisory control circuit in a certain order and the measures are carried out. For this reason,
For example, if there is a failure in the hardware whose monitoring action order is later, the action information for that hardware is read after all the action information for the hardware that does not require the action in the earlier order is read, Will be executed.

On the other hand, in the above fault monitoring device, when the number of monitoring points is large, conventionally, a plurality of processing devices share the monitoring points for processing, but in recent years, the processing capability of processors has increased. Along with this, there is a strong demand for economy and size reduction by allowing a single processing device to perform monitoring processing for a large number of monitoring points. However, when the number of monitoring points increases, the time required for monitoring and taking measures for all monitoring points becomes long, and in particular, when a failure is detected in the hardware with the second highest monitoring order, failure recovery will be delayed. .

Therefore, there is a need for a failure recovery processing method capable of shortening the failure recovery time.

[0007]

2. Description of the Related Art FIG. 6 is a block diagram of a conventional fault monitoring apparatus, and FIG. 7 is a time chart of the conventional fault monitoring apparatus.

In FIG. 6, each hardware 7 displays the presence or absence of a failure at the monitoring point 3, and the processing unit 1
1. The fault monitoring device including the supervisory control circuit 2 and the memory 4 monitors the presence / absence of a fault in the hardware 7 at this monitoring point 3 and takes necessary measures when a fault is detected. The hardware 7 targeted for failure monitoring is a printed board package circuit or the like, but for convenience of description, FIG. 6 shows only eight devices # 1 to # 8. Further, the processing unit 11 constitutes firmware with a microprocessor, RAM, ROM, etc., and the supervisory control circuit 2 is a processing unit.
Often implemented as an LSI circuit for collecting monitoring information and executing measures under the control of 11.

Based on the above, the prior art will be described with reference to FIG. The monitoring processing unit 15 of the processing device 11 periodically gives an instruction to the monitoring control circuit 2 to collect information about a failure of the hardware 7. Upon receiving the above instruction, the monitor control circuit 2 scans the monitor points 3 and causes the monitor information storage area 4a of the memory 4 to sequentially store information indicating the presence / absence of a fault displayed at each monitor point 3 as monitor information. The order of scanning and storage is predetermined, and in this example, the # 1 hardware is always scanned and stored in order.

When the monitoring information starts to be stored in the monitoring information storage area 4a, the monitoring information analysis processing section 15a in the monitoring processing section 15
Determines the measures by sequentially reading and analyzing the monitoring information stored in the monitoring information storage area 4a from the head of the storage area, that is, the storage area in which the monitoring information of the hardware # 1 is stored. Measure content is measure information as measure information transmission processing unit
15b, and the measure information transmission processing unit 15b writes the measure information in the measure information storage area 4b of the memory 4.

This measure information is information indicating that no measure is required for the monitoring point 3 that does not display a fault, and a predetermined measure for the monitoring point 3 that displays a fault, for example, a fault. This is information for instructing disconnection of the hardware 7 or switching to a spare device in the case of a redundant configuration. Here, each hardware 7 is assumed to have a duplicated configuration, and the action information for the monitoring point 3 displaying a failure is information for instructing switching to spare hardware (not shown).

The above-mentioned measure information including the measure-unnecessary information is sequentially written in the measure information storage area 4b of the memory 4, but the order of writing is the same as that of the storage of the monitor information, that is, from # 1 of the monitor point 3. The order is # 8.

When the measure information is written, the supervisory control circuit 2
Moves to the execution of the measure according to the instruction of the monitoring processing unit 15, reads the measure information stored in the measure information storage area 4b of the memory 4 sequentially from the head of the storage area, and is displayed in the measure information for the corresponding monitoring point 3. Take action. When switching to the standby device, the measure information is not sent directly to the monitoring point 3, but the measure information is sent to the switching device (not shown) for switching, but from the convenience of explanation, all measure information is applicable. It shall be sent to point 3. FIG. 7 is a time chart for explaining the above operation. In the figure, an example is shown in which a failure has occurred at monitoring point # 4. Hereinafter, FIG. 7 will be described on the premise of the configuration of FIG.

FIG. 7 shows a state in which the supervisory control circuit 2 is writing supervisory information to the supervisory information storage area 4a of the memory 4 according to an instruction from the processor 11. When the monitoring information is written in the monitoring information storage area 4a, the processing device 11 starts reading the monitoring information. The reading of the monitoring information may be started after all the monitoring information is stored, but in the figure, assuming that the writing speed of the monitoring information is faster than the reading and analysis speed by the processing unit 11, # 1 is used.
5 illustrates an example in which the reading by the processing device 11 is started at the time when the writing of the monitoring information of the monitoring point is finished. Since this operation is performed, the memory 4 of FIG.
An M (Dual Port Random Access Memory) is used so that writing by the supervisory control circuit 2 and reading by the processor 11 can be performed simultaneously.

The monitoring information analysis processing unit 15a of the processing device 11 sequentially analyzes the read monitoring information starting from # 1, and determines the measures for each monitoring point 3. The determined contents, including the case where no action is required, are sent to the action information sending processing unit 15b as action information, and the action information sending processing unit 15b sequentially writes this in the action information storage area 4b of the memory 4. Indicates writing in the measure information transmission processing unit 15b.

If the monitoring information analysis processing unit 15a detects a failure when analyzing the monitoring point # 4, the monitoring information analysis processing unit 15a sends the processing information for the failure to the processing information sending processing unit 15b as described above. Via the action information storage area 4b. Further, when the monitoring information analysis processing unit 15a detects a failure, it stops reading and analysis of subsequent monitoring information and instructs the monitoring control circuit 2 to execute the measure.

If the supervisory control circuit 2 has not yet finished writing the supervisory information at this point, the supervisory control circuit 2 is shown in FIG.
As shown in, the writing of the monitoring information at the monitoring point 3 of # 8 is completed and the measure is executed, and the reading of the measure information is started from the head of the measure information storage area 4b of the memory 4. In this example, no measures are required for the monitoring points # 1 to # 3, but since the measures information for the monitoring points # 1 are sequentially read as shown in FIG. 7, the measures for # 4 are # 1 to # 3. It is executed after the reading of the measure information for the monitoring points of No. 3 is completed.

The time from when the processing unit 11 requests the execution of the measure until the supervisory control circuit 2 starts executing the measure is T ₁ , and the fault is displayed after the supervisory control circuit 2 starts executing the measure. If the time until the reading of the monitoring point # 4 being started is started is T ₂ , the time from when the processing device 11 requests the execution of the measure until the monitoring control circuit 2 starts the measure of the failure monitoring point # 4. Becomes (T ₁ + T ₂ ) as shown in FIG.

FIG. 6 shows the case where the number of monitoring points 3 is small and the scanning order of the failure monitoring points is relatively early. However, the number of monitoring points 3 is large and the scanning order of the hardware 7 in which the failure has occurred is large. In the latter case, the implementation of disability measures will be delayed considerably.

[0020]

In the fault recovery processing method in the fault monitoring apparatus of the prior art, since the action information for the monitoring point is read out in a fixed order regardless of the presence or absence of the fault, the hardware in the subsequent order is affected by the fault. If this occurs, it takes time for the measures to be implemented, and there is a problem that the failure recovery is delayed.

It is an object of the present invention to reduce the time from failure detection to execution of recovery measures.

[0022]

FIG. 1 is a diagram for explaining the principle of the present invention. In the figure, 1 is a processing unit of a fault monitoring device, 7 is a plurality of hardware to be fault monitored, 3 is a plurality of monitoring points for displaying the presence or absence of a fault in each of the plurality of hardware 7, 4 is monitoring information and Storage means for storing measure information, 2
Scans a plurality of monitoring points 3 under the control of the processing device 1 to sequentially store the monitoring information in the storage means 4, and the storage means 4
It is a monitoring / measuring executing means for reading the measure information stored in (1) and executing the measure according to the measure information read to the corresponding monitoring point 3.

Reference numeral 5 sequentially reads the monitoring information stored in the storage means 4 in the processing device 1 to determine the measure for each monitoring point 3, stores the measure content in the storage means 4 as measure information, and then monitors. A monitoring processing unit that causes the measure execution unit 2 to read the measure information and execute the measure.

In the processing device 1, when a notification is received from the monitoring processing means 5 that the monitoring point 3 displaying a failure is detected, the processing information of the monitoring point 3 is sent to the monitoring / measurement executing means 6. 2 is a measure information storage area changing means for designating to the monitor processing means 5 a storage order of measure information different from the storage order of the monitor information so as to be stored in the storage area in the storage means 4 which is read first.

[0025]

In FIG. 1, the monitoring / measuring executing means 2 scans a plurality of monitoring points 3 for indicating the presence / absence of a failure by each of the plurality of hardware 7 under the control of the processing device 1, and sequentially uses the monitoring results as monitoring information. It is stored in the storage means 4. Processor 1
Monitoring processing means 5 sequentially reads the monitoring information stored in the storage means 4, determines the measures for each monitoring point 3 and stores the measure information in the storage means 4, but at that time, the failure is displayed. When the monitoring point 3 is detected, the measure information storage area changing means 6 is notified of this.

Upon receipt of this notification, the measure information storage area changing means 6 monitors the measure information for the monitoring point 3 as described above.
The storage order of the measure information different from the storage order of the monitor information so as to be stored in the measure information storage area (not shown) in the storage unit 4 which is read first by the measure execution unit 2 is designated to the monitoring processing unit 5. As a result, the monitoring processing means 5 stores the measure information in this changed storage area.

When the measure information is stored in the storage means 4, the monitor processing section 5 instructs the monitor / measure executing means 2 to execute the measure. When the monitoring / measure execution means 2 receives this instruction, it sequentially reads the measure information from the storage means 4 in which the measure information is stored and executes the measures in the order in which they were read. Since the measure information is stored in the measure information storage area which the monitoring / measure execution means 2 reads first, the measure information for the monitoring point 3 displaying the failure is first read and executed.

Therefore, in FIG. 1, the recovery process for the hardware in which the failure is detected is executed with the highest priority.

[0029]

FIG. 2 is a block diagram of an embodiment of the present invention, FIG. 3 is a memory state diagram of an embodiment of the present invention, FIG. 4 is a flowchart of an embodiment of the present invention, and FIG. 5 is a time chart of an embodiment of the present invention. .

Throughout the drawings, the same symbols indicate the same objects, 1 is a processing unit mainly composed of firmware composed of a microprocessor, RAM, ROM and the like (above, not shown), and 2 is monitoring / measurement of FIG. In the implementation of the execution means 2, for example, a monitoring control circuit composed of an LSI, 4 is a memory (corresponding to the storage means in FIG. 1), 4a and 4b are a monitoring information storage area and a measure information storage area in the memory 4. . Further, 5a and 5b are a monitoring information analysis processing unit and a measure information transmission processing unit in the monitoring processing unit. For convenience of explanation, FIG. 2 illustrates only eight devices # 1 to # 8 (for example, a circuit of a printed board package) as the hardware 7.

The flow chart of FIG. 4 will be used together with the description of FIG. 2 and FIG. Note that S1 to S16 are symbols for identifying the steps in FIG.

In FIG. 2, the monitoring processor 5 of the processor 1
Activates the supervisory control circuit 2 every predetermined period to activate the supervisory point 3
(S1). When the monitoring control circuit 2 is activated, first, the display content of # 1 of the monitoring point 3 (hereinafter referred to as monitoring point # 1) is stored in the monitoring information storage area 4a of the memory 4 as monitoring information, and thereafter, Storage processing of monitoring information is continued until scanning of all monitoring points 3 is completed (S2 to S3).
. When the storage of all monitoring information is completed, the monitoring control circuit 2
Stands by until the processor 1 gives an instruction to execute a measure (S4).

In the processor 1, when the monitoring control circuit 2 is activated, the monitoring information analysis processing unit 5a starts reading the monitoring information storage area 4a of the memory 4, but the monitoring information of the first monitoring point # 1 is stored. If it does not exist, it is stored and is read (S7, S8). Since the memory 4 is a DPRAM as in the prior art, the monitor control circuit 2
When the monitoring information of the monitoring point # 1 that allows simultaneous writing by the processor and reading by the processing device 1 is stored,
The monitoring information analysis processing unit 5a reads this monitoring information and analyzes whether or not a failure is displayed (S9). If it is not a failure, for example, information of "no action required" is written in the storage area for monitoring point # 1 of the action information storage area 4b, but in reality, nothing may be written in S11 (S10, S11).
If reading of the monitoring information is not completed in this state,
The process returns to the reading of the next monitoring information (S12 → S7). Repeating this, if all the monitoring points 3 do not display the failure, the time counting for the periodic activation (the details are omitted because it is a known technique) is performed and the monitoring is performed when the next activation time comes. Start the control circuit 2 and instruct monitoring (S12 → S
16 → S1).

In the above monitoring information analysis (S9), if it is detected that the monitoring information of the monitoring point # 4 indicates a failure (S10), the monitoring information analysis processing section 5a determines that the monitoring point # 4 Failure detection is performed by the measure information storage area change processing unit 6
Is notified and the subsequent analysis of monitoring information is stopped. Upon receipt of the notification, the measure information storage area change processing unit 6 receives the monitoring point # 4.
The storage order of the measure information so that the measure information of the measure information is stored in the first region of the measure information storage area 4b (the region where the monitor control circuit 2 first reads out, usually the region where the measure information of the monitoring point # 1 is stored). Information indicating the storage area of each measure information, instead of the storage order of monitoring information (hereinafter referred to as address)
Is returned to the monitoring information analysis processing section 5a (S13).

When the monitoring information analysis processing unit 5a receives the above address, it sends it together with the action information to the action information sending processing unit 5b, and the action information sending processing unit 5b specifies this action information in the action information storage area 4b. It is stored in the address (S14).

When this writing is completed, the monitoring processing unit 5 of the processing device 1 activates the monitoring control circuit 2 and gives an instruction to execute the measure (S15). When activated, the monitoring control circuit 2 starts reading from the measure information stored in the first storage area of the measure information storage area 4b. In this case, the monitoring point # 4
Since the measure information of # is read first, the measure specified in the measure information for the monitoring point # 4 (for example, the measure information for instructing to switch the working hardware corresponding to the monitoring point # 4 to the spare hardware). (S5, S
6) Then, the processing in this monitoring cycle is ended.

FIG. 3 is a diagram for explaining the storage states of the monitoring information storage area 4a and the measure information storage area 4b of the memory 4 described above. As described above, assuming that the scanning order of the monitoring points 3 and the measure execution order are the numerical order from # 1 to # 8, the monitoring information storage area 4a stores monitoring information as shown in (1) of FIG. 3, for example. Remembered. The numbers of the illustrated addresses indicate the start addresses of the storage areas in which the monitoring information of each monitoring point is stored.

The measure information is basically stored in the measure information storage area 4b from # 1 in the same numerical order as the monitoring information storage area 4a, as shown in (a) of (2) of FIG. But,
When a fault is detected, for example, when a fault is detected at monitoring point # 4, the measure information storage area change processing unit 6 causes the monitoring point #
The measure information of No. 4 is changed so as to be stored in the first storage area of the monitoring information storage area 4a, that is, the area for storing the measure information of the monitoring point # 1 in (a). The conversion of this address is performed by the following calculation.

First, the start address of the measure information storage area of the monitoring point # 4 is set to the top address "20000" of the measure information storage area 4b. In addition, the monitoring points after the monitoring point # 5, that is, the start addresses of the storage areas for storing the measure information of the monitoring points # 5 to # 8 are from the start addresses of the respective monitoring points in the basic storage state shown in (a). It is set to the value obtained by subtracting 20300 −20000 = 300 from the change in address of monitoring point # 4. Therefore, the start address of monitoring point # 5 is
20400-300 = 20100, and the start addresses of monitoring points # 6 to 8 are 20200, 20300, and 20400, respectively.

Further, the monitoring points having a number smaller than the monitoring point # 4,
That is, the start address of the monitoring points # 1 to # 3 is the value of the address next to the last address of the measure information storage area 4b (not shown in the figure, "20799" is the last address). From 20800 ", the value" 20000 "of the top address of the measure information storage area 4b and the change of the address of the monitoring point # 4 (20
After subtracting (300-20000 = 300), the value of the start address of each monitoring point in the basic storage state shown in (a) is added to calculate. For example, the start address of the monitoring point # 1 is (20800-20000-300) + 20000 = 20500, and the start addresses of the monitoring points # 2 and # 3 are 20600 and 20700. FIG. 3B shows the storage state after the storage area is changed as described above.

Next, the time chart of the embodiment described above will be explained with reference to FIG. 5 and FIG. 5 are the same as and in FIG. In FIG. 5, the monitoring information analysis processing unit 5a
The time when the measures are decided after the analysis of the monitoring information read out by is shown is the same as that of FIG. 7. However, although the measure information is stored in the measure information storage area 4b at the same time as the measure information is determined in FIG. 7, the measure information determined in FIG. Is not established (in both FIG. 5 and FIG. 7, the monitoring information analysis processing unit 5a or 15
The transfer time from a to the measure information analysis processing unit 5b or 15b is ignored).

As described above, in the configuration of FIG. 2, the storage area for the measure information is changed when a failure is detected. Watchpoint #
When the failure is detected by analyzing the monitoring information of No. 4 and the measure is determined, the measure information for the monitoring point # 4 is changed to be stored at the head of the measure information storage area 4b as shown in FIG. At this time, the measure information of the measures # 1 to # 3 (the information indicated by the dotted line in FIG. 5) whose measures have been determined earlier is stored in the storage area as shown in (b) of FIG. Will be moved to the rank. When a failure is detected by the monitoring information of the monitoring point # 4, the analysis of the subsequent monitoring information is stopped as described with reference to FIG. Although it is not actually stored, since the state of not storing is the same as the storage of "no action required", action information for monitoring points # 5 and after is shown in Fig. 5 and Fig. 3 ( It is made to agree with the illustrated content of (b) of 2).

As a result of the above, the measure information for the monitoring point # 4 is stored in the measure information storage area 4 as shown in (b) of FIG.
It is stored at the top of b. In this state, when the monitoring processing unit 5 instructs the monitoring control circuit 2 to execute the measure, the monitoring control circuit 2 starts a process of reading the measure information from the measure information storage area 4b, but at this time, the writing of the monitor information is completed. If not, the process is stopped and the reading of the measure information is started. The reading is performed from the top of the measure information storage area 4b. Since measure information for the monitoring point # 4 is stored at the beginning, the measure for the monitoring point # 4 is performed first.

In FIG. 5, the supervisory control circuit 2 is in a state in which the writing of the supervisory information is not completed when the processor 1 instructs the supervisory control circuit 2 to execute the measure, and the process of reading the measure information is started. It shows a state requiring time T ₁ . However, when the reading of the measure information is started, the measure for the monitoring point # 4 is first executed.
The time from the request of the execution of the measure by 11 to the start of the measure of the fault monitoring point # 4 by the monitor control circuit 2 is only T _1, which is shorter than the conventional technique shown in FIG. 7 by the time T ₂ . . When the number of monitoring points 3 is large, the time T ₂ is large. However, in the present invention, even if the number of monitoring points is increased, the monitoring control circuit 2 causes the monitoring control circuit 2 to execute the measure after the processing device 11 requests the execution of the measures.
There is no increase in the time to start the measures.

Although the embodiments of the present invention have been described with reference to FIGS. 2 to 5, FIGS. 2 to 5 merely show one embodiment of the present invention, and only the embodiments of the present invention are shown. It goes without saying that it is not limited.

For example, in FIG. 4, when no fault is detected, the processor 1 uses the supervisory control circuit 2 for executing the measures.
However, the effect of the present invention does not change even if the monitor control circuit 2 for executing the measure is started regardless of whether or not there is a failure.

Further, in (b) of FIG. 3 (b), after moving the storage area of the measure information for the monitoring point # 4 where the failure is detected to the head, the monitoring points # 5 to # 8 are set to the areas following this. Although stored, the measure information for the monitor point # 4 is stored subsequently to the monitor points # 1 to # 3, that is, the measure information for the monitor point for which the measure is determined prior to the monitor point where the failure is detected. The area following it may be used as a storage area for the measure information of the remaining monitoring points # 5 to # 8. In this case, since the monitoring points # 5 to # 8 are the information of the monitoring points for which the analysis of the monitoring information has not been completed, the measure information storage area 4b is actually used.
However, the effect of the present invention does not change in any case.

Further, in the above description, when the monitoring information analysis processing unit 5a detects a failure, the measure information storage area change processing unit 6 changes the storage area for the monitoring point where the failure is detected by the monitoring information analysis processing unit. 5a, and the monitoring information analysis processing unit 5a sends the changed polar area information and the measures information to the measures information analysis processing unit 5b, but when a failure is detected, the monitoring information analysis processing unit 5a Measure information is sent to the measure information storage area change processing unit 6 together with the notification of detection,
Even if the measure information storage area change processing unit 6 sends the changed storage area information and device information to the measure information transmission processing unit 5b, the effect of the present invention does not change.

[0049]

As described above, according to the present invention,
When a monitoring point that displays the presence or absence of a hardware failure is monitored and a failure is detected, the measures for the monitoring point displaying the failure are executed first, so the number of monitoring points is large.
In addition, even when a failure is detected at a monitoring point that is in the second order in the monitoring order, the failure is quickly recovered.

As described above, the present invention greatly contributes to improving the performance of the fault monitoring device and improving the reliability of the hardware to be monitored.

[Brief description of drawings]

FIG. 1 is an explanatory view of the principle of the present invention.

FIG. 2 is a block diagram of an embodiment of the present invention.

FIG. 3 is a memory storage state diagram according to an embodiment of the present invention.

FIG. 4 is a flowchart of an embodiment of the present invention.

FIG. 5 is a time chart of an example of the present invention.

FIG. 6 is a block diagram of a conventional technique.

FIG. 7 is a time chart of conventional technology.

[Explanation of symbols]

 1 Processing Device 2 Monitoring / Measures Execution Means 3 Monitoring Points 4 Storage Means 5 Monitoring Processing Means 6 Measures Information Storage Area Changing Means 7 Hardware

Claims (1)

[Claims] 1. Under the control of the processing unit (1), the monitoring / measure execution means (2) scans a plurality of monitoring points (3), each of which has a plurality of hardware (7) and displays the presence / absence of a failure. Watchpoint (3)
When the display contents of are stored in the storage means (4) as monitoring information sequentially, the monitoring processing means (5) in the processing device (1) sequentially reads the monitoring information and takes measures for each monitoring point (3). A fault recovery process in a fault monitoring device that determines and stores the action content as the action information in the storage unit (4), and then causes the monitoring and action execution unit (2) to read the action information and execute the action. A method for receiving a monitoring point (3) in the processing device (1), when the monitoring processing means (5) receives a notification that a monitoring point (3) displaying a fault is detected. The monitoring processing means (5) has a storage order of measure information different from the storage order of the monitor information so that the measure information is stored in a storage area in the storage means (4) which is read first by the monitor / measure execution means (2). It is equipped with a measure information storage area changing means (6) to specify the failure of the hardware (7). A failure recovery processing method characterized by preferentially executing measures for the failed hardware (7) when detected.