| Case Name |
Mizuho Financial Group Banking System Failure |
| Pictograph |
|
| Date |
April 1, 2002 |
| Place |
Japan |
| Location |
Mizuho Bank |
| Overview |
The serious banking system problems disrupted the new banks' operations following the April 1 launch of Mizuho Bank and Mizuho Corporate Bank into which Dai-ichi Kangyo Bank Ltd., Fuji Bank Ltd. and Industrial Bank of Japan Ltd. had been consolidated under the management of Mizuho Holding in Tokyo. The Mizuho Bank's important first day in business was tarnished by the ATM (automatic teller machine) transaction errors and delayed automatic debits for utility and credit card bills. The biggest banking system failure in history caused chaos, from which the recovery took more than one month until the bank's operations went completely back to normal. A long period of failed discussion among the three banks about whose computer systems would survive the merger left insufficient time for the implementation period of a relay system to bridge the banking systems. The bank group went ahead with integration and the operation of the unified banking system, despite the testing period of the unified operation was not long enough for the large-scale operation. |
| Incident |
As part of the finance-industry reorganization, Dai-ichi Kangyo Bank Ltd., Fuji Bank Ltd. and Industrial Bank of Japan Ltd. were consolidated into Mizuho Bank Ltd. and Mizuho Corporate Bank Ltd. under the management of Mizuho Holding in Tokyo. The new Mizuho Bank's first day in business (April 1, 2002) was tarnished by the ATM (automatic teller machine) transaction errors in which debits were made from accounts although customers did not receive cash from the machine, as well as unprocessed scheduled automatic debits for utility and credit card bills. The banking system failed to process 105,000 automatic debits scheduled for April 1. Delayed debits backlogged, and the number of delayed debits reached 2.5 million by the end of April 5. An additional 30,000 double debits were found between April 1 and April 5. The biggest banking system failure in history caused chaos, from which the recovery took more than one month until the bank's operations went completely back to normal. |
| Sequence |
The course of this case is summerized in list1. |
| Cause |
(1) Technical factor (direct cause)
a. ATM transaction errors
The Mizuho Bank's primary banking system (previous Dai-ichi Kangyo Bank system using the Fujitsu mainframe) was not modified properly.
The program added to handle logical connection between the primary system and the relay systems had a bug that caused a specific problem in certain circumstances. It was an oversight during the program development.
b. Delayed automatic debits
- Invalid information in the transaction request data.
- A bug in the program developed for processing debits transaction for bridged banking network.
The problems surfaced only under certain conditions such as "data of different types or categories are combined in a certain manner during processing" when "data overflow or excessive transactions accumulated in the data storage area for the relay computer".
The first bug fix failed to correct the programming errors and caused the recurrence of the online banking system failure on April 8.
c. Insufficient testing period of the unified operation and load tests.
Program development always accompany bugs. Thorough operational testing and debugging were critical for the unified system before bringing them on-line. Testing should have simulated the real large scale banking operation and various complicated situations.
(2) Corporative factor (principal cause)
a. Delayed decision on the system integration scheme (problems during planning)
The merger lacked the dominating party that can effectively take charge, and the three banks simply could not agree on whose computer systems would survive the merger. The system integration scheme changed a few times until the three banks reached a settlement.
The three banks should have proposed the most adequate scheme after assessing the requirements on the unified banking system and evaluating the existing banking systems of the three banks. Instead of determining the system that would survive the merger, the three banks knocked out a compromise, involving relay systems to simply bridge the existing systems of the three banks. This temporizing scheme involving relay systems was not inadequate itself; however, delay in decision making resulted in insufficient period for testing and systems integration.
The scheme lacked the architecture concept. When making changes in banking systems such as integration, it is best to choose the time of year when processing volume is smaller. April 1 has been usually the busiest day of the year for the three banks, and scheduling system changes on such a day was not advisable. In addition, the three banks that merged into Mizuho were too fixated on their schedule to perfect the systems integration, which compounded the problem.
b. Delay in preparations for system integration
Problematic management organization for the integration project.
The Mizuho Holding management committee did not receive accurate status reports in a timely manner, while there had been evident delays in preparation for the scheduled system integration. The three banks independently performed preparation towards the scheduled system integration without reporting status to each other or to the holding company.
The CEOs (chief executive officer) of the three banks knew of the delay after receiving several reports in 2001 from their banking system divisions. The CEOs simply instructed their banking system divisions to make up for the delay without looking into details.
c. The bridging systems were not tested thoroughly for their performance and processing loads before bringing them on-line.
d. The Mizuho Holding management committee went ahead with integration and the operation of the unified banking system, despite the delay in preparation and risks of system failures during the real banking operation. |
| Response |
(1) FSA dispatched the financial regulators to inspect Mizuho Bank, and the Bank of Japan inspected Mizuho Holding and Mizuho Corporate Bank on-site. The inspections took a month to investigate the cause of banking system failures and reviewed the handling of problems as well as the banks' measures for preventing recurrences of system failures.
(2) FSA issued a business improvement administrative order to Mizuho Holding.
The order includes measures for preventing system failures and responses, as well as improvement in administration and unified banking system to help preventing recurrences of system failures and clarifying management's responsibility.
Because of the estimated large sum of development cost for the unified banking system and risks of system failures during the development, it called for drastic prevention measures such as improvement in management of system development and administration, reorganization of the banking system division, reinforcement of the auditing system, active involvement of the management committee in the system development project, better communication and information sharing among the three banks.
FSA also ordered to create guidelines that enable prompt responses and adequate resolutions to system failures. Although Mizuho had guidelines for actions in case of system failures prior to the merger in April, they lacked customer-oriented instructions of how to respond to individuals and corporate customers and announce schedules of restoration. |
| Countermeasures |
(1) In response to the system failures, Mizuho formed the system failure special committee that includes outside consultants.
(2) Establishment of corporate governance system
Rivalry and power struggle among the three banks resulted in delay of the schedule and contributed to the system failures. Such organizational issues must be eliminated to construct efficient management systems.
Mizuho proposed reorganization of its administrative structure to consolidate sound and efficient management.
Some management officers of the three banks was to be let go to make the reporting system efficient so that Mizuho Holding can control the banks' operations.
The management committees of the new banks were previously structured to include equal number of officers from the three banks as a compromise to the rivalry and power struggle among the three banks, which contributed to the inadequate business practice. Mizuho planned to restructure the organization after reviewing the administrative structure as well as the development scheme of the unified banking system.
(3) The three banks decided not to independently prepare and develop banking systems prior to integration, which lead to an absence of exchanging development status among three banks and contributed to system failures. The new development scheme encouraged cooperation of the engineers of the three banks to jointly develop the integrated banking system. |
| Knowledge Comment |
(1) System failures generally occur on the systems that bridge external systems to the mainframe.
(2) Systems require thorough tests including connection, stress tests, enhancement tests, and fault tolerance before bringing them into the real operation.
(3) A system design requires a clear architecture concept to begin with.
(4) Integration of systems for merger requires detailed review of the existing systems to propose the most rational approach and scheme.
(5) The administration executives must have better understandings on the significance of mission-critical systems for business operations.
(6) Reorganization and rationalization of the executive management is essential for successful corporate merger. It is irrational in business perspective to keep all executives and employees or to form the management committee with the same number of executives from companies involved in the merger. |
| Background |
Starting with the Bank of Tokyo-Mitsubishi UFJ, there had been a series of incidents of system failures at financial institutions such as Mizuho Bank, Post Office Savings Bank and IY Bank (current Seven Bank, Ltd.). The system failures surfaced to be ATM transaction errors resulted from problems in the bridge connection between banking network systems and the mainframe.
There were also social factors that contributed to these incidents: diversification of services and radical reorganizations in the finance industry observed in the recent years. Merger of banks as we have seen in the Bank of Tokyo-Mitsubishi UFJ and Mizuho Bank requires significant revisions of banking systems. Such banking systems must also offer capabilities of debit transactions and networking with Post Office Savings Bank and credit unions, which significantly increase the system load.
Electronic transactions offer direct business between sellers and buyers without an intervening party. Such transactions require a high availability transaction processing solution commonly developed on an open system environment, rather than on a legacy system environment. While such a solution offers high availability and consistent, short response times to enable online real-time processing of mass transaction data, it requires relatively complex network architecture. Many corporations and manufacturers now face difficulties in catching up with e-commerce trends and implementing new transaction solutions.
Unquestioning belief in information systems is another problematic factor observed commonly in government agencies and industries in Japan. Unlike the quick economic recovery of the IT industry in the U.S., declining stock prices of many IT companies in Japan has been putting a damper on already faltering investment since the collapse of the asset-inflated economy in 1990's. Many IT companies are still struggling with sluggish corporate growth without being able to alleviate their staffing shortage. While many companies brought new IT technologies to their work environment, an inadequate employee training is preventing them to take full advantage of their updated information systems. Some of the leading manufacturers recently announced large layoffs of more than 90,000 employees; however, most of the layoffs are targeting those who joined the companies through the mass hiring during the boom of home appliances, not the ones who joined them after the coming of age of the Internet. This also proves that corporations have a serious shortage of employees who are familiar with the recent information technologies. |
| Incidental Discussion |
Although the consolidation of the tree banks created the world's largest bank by assets and integrating the three banks' systems required enormous tasks, this incident revealed poor corporate management and excessive optimism over the new business and the mission-critical system integration, which may also be observed across industries. It is critical for banking business to build a fault-free computer system. In particular, fault-free banking information and account systems are mission-critical to the bank. It is unavoidable for any merger of banks to unify such mission-critical systems. Mizuho group had been preparing for the scheduled system integration on April 1, 2002 since the previous year; however, the unified banking system was developed through a trial and error process to which the cause of system failures was attributed.
The management committee of the new banks multiplied in number after merger by keeping all members of the three banks, a practice commonly observed in any mergers. Without having the dominating party that can effectively take charge, all administrative decisions required unanimous agreement of the three banks. Rivalry and power struggle among the three banks resulted in delayed decision on the system integration scheme and one-year postponement of the original integration plan. This delay ended up leaving insufficient time for performance testing of bridging systems, data preparation and load testing for the large-scale operation. While three banks had evident delay in preparation for the system integration, they were too fixated on their schedule in April to allow the Mizuho Holding management committee making appropriate judgments on the delay. Corporative factors including poor management and communication greatly contributed to the incident. Mizuho group bears heavy responsibility as the banking group that caused general chaos involving thousands transaction errors and delayed debits.
This incident caused immeasurable damage to the society and showed problems of the information society. It also proved how much benefit the Japanese economy receives from information technology (IT) and made one reaffirm the significance of it in the society. System failures have also been reported in the finance and the aviation industries. The pressing need for thorough inspection and maintenance of information systems are obvious.
Since this incident led to potential crisis of confidence in the Japanese financial system and the financial administration, Japan needs to emphasize significance of information systems and encourage investment and education to the IT industry as well as to the organizations that utilize the technology. |
| Scenario |
| Primary Scenario
|
Poor Value Perception, Difference in Culture, Poor Concept, Poor Organization, Insufficient Analysis or Research, Insufficient Prior Research, Usage, Operation/Use, Malfunction, Poor System, Loss to Organization, Social Loss
|
|
| Sources |
"Jibun-o Mamoru Kiki Kanri Jyutu (Risk Management for Saving Yourself)", Nikkei Business Associe, July 2002, Vol. 3, Nikkei Business Publications Inc., 24-26.
"Dai-Shippai-no Mechanism, San-no Baisuu-no Ronri (Mechanism of Catastrophe, Theory of Three Multiples)", President, July 1, 2002, 88-93.
"Mizuho Gin, Teimei-no Nisyuukan-o Ou (Doomed 2 Weeks at Mizuho Bank)", Nikkei Computer, April 22, 2002, 12-15.
"Scoop - Mizuho Ginkoo-no System Syoogai, Shinsoo-ga Hanmei, Taigai Setuzoku-kei-no Syuusei-o Miss, Furikae Made (News Flash: Mizuho Bank's System Failures Revealed, Bug in Network Bridge Program Interrupted Debits)", Nikkei Computer, May 06, 2002, 11.
"Joohoo System-no Kiki Kanri-o Minao-se (Reviewing Plan of Risk Management for Information Systems)", Nikkei Computer, June 03, 2002, 154-160.
|
| Multimedia Files |
Figure1.Banking System Integration Scheme
|
|
List1.course of this case
|
| Field |
Mechanical Engineering
|
| Author |
NAKAO, Masayuki (Institute of Engineering Innovation, School of Engineering, The University of Tokyo)
|
|
