Home Jp Yutaka Matsuno's Homepage dsn2012FMHSKI

全文

(1)DS-Bench Toolset: Tools for Dependability Benchmarking with Simulation and Assurance Hajime Fujita⇤ , Yutaka Matsuno⇤§ , Toshihiro Hanawa† , Mitsuhisa Sato† , Shinpei Kato‡¶ , and Yutaka Ishikawa⇤ ⇤ The University of Tokyo, Japan, † University of Tsukuba, Japan, ‡ University of California, Santa Cruz, USA {hfujita, matsu}@cc.u-tokyo.ac.jp, {hanawa, msato}@cs.tsukuba.ac.jp, [email protected], [email protected]. Abstract—Today’s information systems have become large and complex because they must interact with each other via networks. This makes testing and assuring the dependability of systems much more difficult than ever before. DS-Bench Toolset has been developed to address this issue, and it includes D-Case Editor, DS-Bench, and D-Cloud. D-Case Editor is an assurance case editor. It makes a tool chain with DS-Bench and D-Cloud, and exploits the test results as evidences of the dependability of the system. DS-Bench manages dependability benchmarking tools and anomaly loads according to benchmarking scenarios. D-Cloud is a test environment for performing rapid system tests controlled by DS-Bench. It combines both a cluster of real machines for performance-accurate benchmarks and a cloud computing environment as a group of virtual machines for exhaustive function testing with a fault-injection facility. DSBench Toolset enables us to test systems satisfactorily and to explain the dependability of the systems to the stakeholders. Keywords-dependability benchmarking; assurance case; fault injection; distributed systems; virtual machine. I. I NTRODUCTION Information systems have become much more important to our daily life, as these systems are ubiquitous in our society. They are used everywhere, for instance, in transaction systems and embedded systems. These systems provide very important infrastructures, and unexpected suspension of the services provided by these systems costs a lot. Therefore these systems should be dependable [1]. Although highly dependable systems, such as highavailability servers, are likely to form parallel and distributed systems in order to acquire redundancy, the tests of such large-scale systems are troublesome. Exhaustive testing under realistic conditions is getting more and more costly and time-consuming, particularly in terms of setting up of the test environment. In addition, a highly dependable system should be equipped with a combination of multiple functions for fault tolerance against overload, hardware faults, etc. Thus, a test support tool, one that configures test environments automatically and executes various kinds of tests in a sophisticated manner, is urgently needed. In such tools, any anomalies, including hardware faults, § Currently he is in Nagoya University, Japan. ¶ Currently he is in Nagoya University, Japan. This research was done while he was in The University of Tokyo.. unpredictable overloads, operational failure due to human error, should be treated flexibly, and in a unified manner, similar to the stress of ordinary workloads. Furthermore, as systems become more networked and involve many stakeholders, the developers or suppliers of the systems need to be able to explain the dependability of the systems, even to the end users. Proof of the dependability of the systems is supported mostly by the results of various methods such as testing, verification, benchmarking, etc. However, it is sometimes difficult to assure users that the system can be certified to be dependable based on results obtained by such methods. Therefore, support tools for assuring dependability based on observable test results are desired. To address these problems, we have developed the following tools: A benchmark test framework for system dependability, referred to as DS-Bench, a description and agreement support tool, called D-Case Editor, and an execution environment, D-Cloud. D-Case Editor makes a tool chain with DS-Bench, and exploits the test results as evidences of the dependability of the system. DS-Bench measures various dependability metrics according to a specific scenario. DCloud supports DS-Bench so that it can perform rapid system tests. The suite formed by these support tools is called DS-Bench Toolset. DS-Bench Toolset provides the following features. • DS-Bench Toolset obtains dependability metrics on the target system using various benchmarks under several anomaly situations, and • it performs benchmark tests using the whole target system, including operating system, network, etc., on a hybrid environment of both physical and virtual machines, and • it provides the evidence for the assurance case based on the benchmark results. In the present paper, we demonstrate that DS-Bench Toolset works as a benchmark for a practical distributed system with fault injection. II. R ELATED W ORK DBench [2] is a project aimed at establishing dependability benchmarking methods for several domains. The.

(2) project has defined several concepts essential for dependability benchmarking. Among of them, “faultload” represents a set of faults and exceptional events based on operator faults, software faults, and hardware faults [3], [4]. However, faultload cannot emulate undesired overloads such as interventions by other applications. In fact, since networked and distributed systems can usually be easily affected by external influences, a unified manner in which to specify such workloads and faultloads is required. However, the project did not provide a total software framework to integrate multiple dependability benchmarks. As a large-scale software test platform, ETICS [5] provides automated test environments for grid and distributed software on a grid computing platform using Condor as a workload management system. Since ETICS handles tests on the service level, ETICS cannot test the dependability of a system including the OS layer. On the other hand, a number of fault injection techniques for program tests have been proposed. DOCTOR [6] is a software fault injector, that supports memory faults, CPU faults, and communication faults, without modification of the source code to be tested. Xception [7] uses the kernel module to inject the fault, and the tester sets the hardware breakpoint to a specified address. When the process reaches the breakpoint, the fault is injected into the register or memory as an incorrect value. BOND [8] uses a special agent to R intercept the system call of the Windows NT!version 4.0. The agent hooks the system call from the target application and returns an incorrect value to the application. The above tools are categorized as software fault injection (SWIFI) tools. However, essential to such tools, source code must be modified, or a specific environment, such as OS with a special driver on a specific version, is required. FAUmachine [9] performs a software test using virtual machines as the fault injection mechanism. However, since FAUmachine does not provide an automated test environment, the tester must configure the test environment manually. NFTAPE [10] is a fault injection framework for distributed systems. NFTAPE has component-based architecture including a control host for managing the test and fault injection, and a process manager on each target host, which runs the fault injector, workload generators, target applications, and so on. In NFTAPE, lightweight fault injectors (LWFIs) are used in order to provide any fault injection method needed, and automatic testing is available. However, an LWFI uses a special API as a trigger event. Therefore, existing and commonly-used tools are not directly applicable for NFTAPE since they must be modified before being suitable for NFTAPE. Moreover, similar to the above tools, system-wide tests, including entire OSs and networks, cannot be performed. System assurance has become of great importance in many industrial sectors. Safety cases (assurance cases for the. safety of systems) are required for submission to certification bodies for developing and operating safety-critical systems, e. g., automotive, railway, defense, nuclear plants, and sea oil production platforms. There are several standards, e. g. the Rail Yellow Book [11] and MoD Defence Standard 00-56, which mandate the use of safety cases. There are several definitions for assurance cases [12]. We show one such definition below [13]. “a documented body of evidence that provides a convincing and valid argument that a system is adequately dependable for a given application in a given environment.” Assurance cases are often written in structured documents using a graphical notation to ease the difficulty of writing and certifying them. Goal Structuring Notation (GSN) is one such notation standard [14]. Currently there are few tools for assurance cases. One notable tool is ASCE tool [13], which is widely used in several areas such as defense, safety-critical areas, and medical devices. However, the current ASCE tool is mostly used to generate documents for certification, and not so much used during the various development phases. Our intention is to use assurance cases for arguing for the dependability of the systems when developing and testing them, as studied in [15]. Furthermore, as far as we know, in the benchmarking and fault injection communities, there has not been much discussion on assuring how the test results contribute to the dependability of the target systems, especially using certain frameworks, such as assurance cases. This seems because the contribution of test results to dependability is taken for granted in these communities. However, as systems become large and complex, this becomes less clear, in particular for non-experts. III. DS-B ENCH T OOLSET In order to support systematic measurement of several dependability metrics, DS-Bench, a tool for dependability benchmarking, and D-Cloud, a test environment for performing rapid system tests, have been developed. D-Case Editor, which is a support tool for discussions between users and developers, has also been developed [16]. We are enhancing D-Case Editor in order to allow it to collaborate with DSBench. Figure 1 illustrates an overview of DS-Bench Toolset. Each DS-Bench system has one DS-Bench controller with benchmark database, and one or more benchmark target machines. The target machines can be either physical machines or virtual machines. Allocation of physical machines and creation of virtual machines are managed by the DCloud controller. A user interacts with the toolset via D-Case Editor or DS-Bench controller’s web interface. DS-Bench controller also controls benchmark procedures. Benchmark programs are executed on benchmark target machines. On each target machine there is a small server program to.

(3) receive commands from the controller. The toolset is capable of simulating anomaly states in the target systems by injecting anomaly loads. Anomaly loads are generated by anomaly generators. Benchmark programs and anomaly loads are combined into benchmark scenarios. Benchmark scenarios, benchmark programs, anomaly generators, and benchmark results are stored in the benchmark database. We define several types of protocols described by XML between the D-Case Editor and the DS-Bench controller, and between the DS-Bench controller and the D-Cloud controller. Tester DS-Bench Controller Benchmark Databases Anomaly Anomaly A nomaly Benchmark Script Script Scenario. D-Case Editor D-Cloud Controller. Target Machines (Physical). Figure 1.. Benchmark Benchmark Results Benchmark Results Results. Anomaly Anomaly A nomaly Anomaly Generator Generator G enerator Generator. Benchmark Programs. Target Machines (Virtual,Cloud). Overview of DS-Bench Toolset. A. Assurance Case Editor with Parameters: D-Case Editor To make assurance cases more useful in system development, D-Case Editor, a free assurance case editor has been implemented as an EclipseTM plug-in using Eclipse GMF. D-Case Editor supports GSN, and provides the following features: a GSN pattern library function and a prototype type checking function [17], a consistency checking function using an advanced proof assistant tool, and the formation of a tool chain with the development tools. The tool chain between DS-Bench and D-Case Editor is designed as follows. •. •. Figure 2.. D-Case Editor Snapshot when setting parameters. PP Performance Performance Benchmark Benchmark Benchmark Programs. DS-Bench and D-Case Editor share the same XML format for test results. D-Case Editor can set parameters for DS-Bench scenarios, such as latency and the access rate of web server systems. Then D-Case Editor directly invokes DSBench and obtain the test result (described in Section III-D).. Figure 2 shows a snapshot of D-Case Editor, when setting the parameters of a DS-Bench scenario. Assurance cases edited in D-Case Editor are called “D-Case Diagrams” (sometimes just “D-Case”).. B. Dependable System Benchmark Framework: DS-Bench DS-Bench is a framework for conducting multiple dependability benchmark tests. The details of DS-Bench are described below. 1) Supports Different Benchmark Programs: DS-Bench itself is not a measurement tool for dependability metrics. Instead, DS-Bench supports execution of many kinds of programs. A benchmark program may be an existing one already available in the market, or one that is created by a user of the target system for a very specific purpose. Therefore, DS-Bench supports a wide variety of programs. Moreover, new programs can be added to the benchmark programs to meet the user needs. There are two challenges in supporting multiple, arbitrary programs. First, each benchmark program has different ways of specifying benchmark parameters. This can be an issue, especially because we provide a common graphical user interface for specifying benchmark parameters. The second problem involves how to store the result. When a benchmark completes, it displays its results on screen. Usually this output is formatted in a text table so that a human user can easily recognize the result. However, while these tables are human-friendly, they are not computer-friendly. They are not suitable for being recorded as data, especially when we want to reuse the data, for instance, in making charts, comparing results between different runs, and so on. In order to solve these issues, DS-Bench introduces a benchmark description for each benchmark program. The benchmark description describes what kind of parameters the program takes, and which part of the output should be interpreted as data. When a new program is added to the DS-Bench framework, a new benchmark description should be prepared. Figure 3 shows an example of a benchmark description for iperf, a network bandwidth measurement tool. Figure 4 shows an input dialog for the iperf benchmark..

(4)

(5)

(6) !

(7) "# $

(8)

(9) %

(10) & ' ( ) & & & ) (

(11) & & ' (

(12) ' ( ) & & ) ) (!

(13) & () ) ( 9 . & )

(14) (!

(15) & & ) (!

(16) & ) (!

(17) & *"(

(18) % ) (

(19) &

(20) ' ( ((

(21)

(22) +,+- .+/+- &+-0 +-1 $ .2 &20 21 $ 2 "+3%+-"-

(23) %+-"- ,456/1! %+-"- -% ) +,+- .+/+- &+-0 +-1 $ ) ! ! :(( ;

(24) <( &( &( (( )

(25) 5 .$ ) <( ! 1 $ ! ) 7 828) 7 )

(26) . Figure 3.. An Example of Benchmark Description (iperf). Figure 4.. An Example of Benchmark Settings. This dialog is automatically generated by DS-Bench by interpreting the benchmark description shown in Figure 3. Benchmark programs are stored in the benchmark database and downloaded by each target machine, on demand. In the current implementation, benchmark programs R packages (.deb) and fetched by are prepared as Debian! the apt tool. 2) Benchmark Results: When a benchmark program completes, the results are sent to the controller. The results are automatically extracted from the raw text output of the.

(27) '"

(28) ( ) !"

(29) #$ %&'

(30) ()*& ( + ()* %,-',.

(31)

(32) "

(33) + %&' * /&/01 %,-',.

(34)

(35) "

(36) + %&' * /&&01 . *

(37) + , .

(38)

(39)

(40)

(41)

(42)

(43)

(44)

(45)

(46)

(47) !

(48)

(49)

(50) !"#

(51) $%$ #

(52)

(53)

(54)

(55)

(56) &

(57)

(58)

(59)

(60)

(61) !

(62)

(63)

(64) !"#

(65) $%% #

(66) .

(67) . . Figure 5.. An Example of Data Extraction. program according to the rules specified in the benchmark description. Figure 5 shows an example of data extraction done by DS-Bench. Results are stored in the benchmark database so that they can be referenced later. DS-Bench supports several different ways to examine the results. Results can be plotted as graphs, or printed as a table. Another data processing method supported is an automatic data reduction. DS-Bench supports multiple target machines in a single benchmark run, so there may be more than one result for some value. DS-Bench applies one of a number of available “data reduction” operations on these values. For example, the average score of results from all target machines can be calculated and displayed. 3) Anomaly Loads: DS-Bench supports several anomaly generators to generate anomaly loads. Anomaly generators are basically classified into two categories. The first type of anomaly generator consists of programs that run on target machines. For example, there are programs that consume a lot of computing resources, such as CPU time, memory, and I/O bandwidth, to simulate overloaded circumstances. Some benchmark programs can be used for performance measurement as well as for anomaly generation which puts stress on the system. Benchmark programs and anomaly generators together are registered in the DS-Bench framework and users may indicate whether a program is executed as an anomaly load. This type of anomaly generator can be easily added to the framework in the same way as regular benchmark programs. The second type of anomaly generator injects a fault from outside of the target machines, such as the cutting of an external power source. In addition, several fault injection methods that simulate hardware malfunctions, such as a memory bit flip, a disk I/O error, or a network packet drop, are supported by FaultVM, a virtual machine monitor with fault injection features. FaultVM is described later in Section III-C. These fault injectors are also used as anomaly generators. These anomaly loads are generated by.

(68) the D-Cloud controller. 4) Benchmark Scenarios: The most important feature of DS-Bench is that it supports the making and execution of benchmark scenarios, which describe when to run which benchmark program or anomaly load. Benchmark programs and anomaly loads can be overlapped in their execution. C. The Execution Platform for DS-Bench: D-Cloud D-Cloud is an execution platform for DS-Bench using both a cluster of physical machines and a cloud computing environment. The details of D-Cloud are as follows. 1) Overview of D-Cloud: D-Cloud was proposed as a large-scale software testing environment, using cloud computing technology for dependable distributed systems [18], [19]. Originally, D-Cloud provided a cloud service based on “Infrastructure as a Service (IaaS),” including a virtual machine environment to develop a dependable system and to assure that the system, in fact, is dependable. It was also supposed to allow the test procedures to be automated as a scenario using a large amount of computing resources in the cloud by interpreting system configurations and test scenario written in XML, as well as enable tests, including hardware faults, by emulating hardware faults by FaultVM flexibly. In addition, FaultVM-SpecC was proposed in order to enable flexible customization of FaultVM, and provide a model where the virtual machine is integrated with the device, and is described in SpecC language [20]. We introduce D-Cloud as an execution platform for the DS-Bench Toolset with several modifications and extensions. In this case, the test scenario should be provided by the DSBench controller instead of the proprietary scenario only for D-Cloud. In order to measure the actual performance by using DS-Bench, physical machines should also be included in D-Cloud, and the resource management facility for the physical machine environment should be added to D-Cloud. 2) Structure and Management of Computing Resources: As shown in Figure 1, D-Cloud includes both a physical machine environment and a virtual machine (VM) environment as a private cloud. In addition, there is a controller node, which controls all of the physical machines and VMs, namely the D-Cloud controller. The D-Cloud controller first gives a list of available resources to the DS-Bench controller, and then it assigns several physical machines and/or guest OSs provided by cloud management software to the specific benchmark test according to the request from the DS-Bench controller. 3) Physical Machine Environment: This environment is used for testing those characteristics that can be only revealed by the actual hardware environment, such as performance evaluation, realtimeness, and timing errors. Anomaly behavior caused by software problems, such as software bugs or overloads, and hardware malfunctions that can be caused by manipulating hardware devices other than computer components, such as network disconnections and. '

(69) '

(70)

(71) #%

(72)

(73) '('

(74) ).

(75) .

(76)

(77) . . . & . !

(78) . . "

(79) # $

(80) %

(81) . & "

(82) . Figure 6.. Workflow using D-Case Editor and DS-Bench. power shutdowns, are also simulated using physical machines. In the physical machine environment, network down and power off tests are performed using a network switch and a power distribution unit (PDU) which is capable of being switched on/off remotely using the Simple Network Management Protocol (SNMP), or a machine halt using the Intelligent Platform Management Interface (IPMI) protocol. 4) Cloud and VM Environment: When dealing with faults in memory flips, or with I/O devices, etc., such faults can be simulated on virtual machines. If the system requires many computers and it is difficult to prepare the required number of physical machines, virtual machines may also be used. We have adopted the OpenStackTM platform for managing massive Virtual Machines since OpenStack has become the de-facto standard for open-source cloud management software [21]. OpenStack is written mainly in Python, and we can easily modify the source code on-the-fly, if needed. A cloud computing environment consists of one or more cloud-controller node(s) and multiple VM nodes. The cloud controller node manages various kinds of OS system images that run on virtual machines. The controller transfers an OS system image to each VM node to execute the OS on the Virtual Machine Monitor in the VM node. A VM node also receives commands for fault injection based on the fault scenario, and injects a series of faults into the Virtual Machine Monitor. D. DS-Bench Results as Evidence for D-Case DS-Bench is designed to be a part of a system development process assured by D-Case. In particular, DS-Bench provides evidence that systems actually meet requirements written in D-Case. In order to support this workflow, D-Case Editor and DSBench are designed so that D-Case Editor can import the results from DS-Bench as evidence for D-Case. A basic.

(83) Table I S PECIFICATIONS OF D EMONSTRATION E NVIRONMENT # of Nodes CPU Memory OS Network Switch Cloud SW VM. Virtual Machines Democlient. 6 (Controller+VM x1, VM x2, Phys x3) R Xeon"processor R Intel" E5506 x2 (8 cores) 24GB UbuntuTM 10.04 DellTM PowerConnectTM 5324 OpenStack Diablo (VLAN Manager) KVM 0.14.0. Democlient. Single IP Address Fault (Linkdown). Switch. HTTP. HTTP. Tammie06. Tammie07. Wiki files. workflow using the toolchain is shown in Figure 6 and described below. First, stakeholders of the system being developed discuss the dependability requirements of the system, then record the conclusion as a D-Case diagram using D-Case Editor (Figure 8). The diagram may contain some quantitative requirements, such as performance requirements (e.g. “the server shall process the request at a rate of more than 250req/s”) or availability requirements (e.g. “the downtime of the system shall be shorter than 5 seconds under one node failure”). Then DS-Bench is used to measure these quantitative scores from the system directly. A D-Case Editor user chooses and imports a benchmark scenario suitable for the current situation, from the benchmark database of DS-Bench. If no such scenario is found in the database, a user may create a new one. Then the user may modify some of the parameters defined in the scenario according to D-Case. For example, in some scenario, the request rate from a client machine may be modified. In addition, the user declares the requirements for the result of the scenario, such as the maximum latency of all requests. After that, the user runs the benchmark test from DCase Editor. D-Case Editor then requests DS-Bench to execute the benchmark test. When the test completes, DCase Editor retrieves the result to show whether the result meets the requirement. A new node, called an evidence node, is added to the leaf of the D-Case diagram, which represents the benchmark result as evidence supporting the successful meeting of the requirement. If the requirement is met, the evidence node is shown as a blue node. Otherwise, it is shown as a red node. A link to information on the details of the benchmark is recorded in the evidence node (Figure 8). IV. D EMONSTRATION OF DS-B ENCH T OOLSET This section describes a demonstration of DS-Bench Toolset. A typical web server-client system is introduced, then it is shown how DS-Bench Toolset can be used with the example system to improve dependability. A. Target System Figure 7 shows an overview of the target demonstration system. The system simulates a typical web server system that hosts the MoinMoin Wiki system [22].. Physical File server Machines. Figure 7.. Target System for Demonstration of DS-Bench Toolset. The server system consists of front-end web servers with an Apache HTTP Server, and a shared file server. Multiple client machines concurrently access the server. In this demonstration, the front-end web servers are prepared as physical machines, and the client machines are prepared as virtual machines. This is because it is crucial to measure the server performance using the actual hardware, while it is much more important for clients to be deployed easily and require less physical hardware resources. This system has two web server nodes, and two nodes form a single IP address web server cluster using the SSPA (Speculative SYN Packet Acceptance) mechanism [23]. By using the SSPA mechanism, all traffic from clients to the server is broadcasted to both of the server nodes. New connection requests from clients are distributed to both of the server nodes. If one server node is not responding, the other server node responds to the client immediately. Table I indicates the specifications of the hardware and software environments for this target demonstration system. B. Test Example In this demonstration, we focus on a web server failure case. The web server failure is simulated by shutting down the network link by controlling a network switch connected to the target physical server. Figure 8 shows a part of the D-Case diagram prepared for this server system. This sub tree illustrates a discussion on a failure that one server fails during the benchmark test. Of course, there should be many other branches in the diagram, which discuss other kinds of faults. However they are omitted because space is limited. Figure 9 is a screenshot of the benchmark scenario editing mode of DS-Bench. Each horizontal line represents a timeline for scheduled jobs (benchmarks and anomaly loads) for each target machine. In this figure we see three target machines, from the top, server node 1, client machine 1, and client machine 2. We do not discuss server machine 2 on the timeline because we do not execute any benchmark.

(84) test is executed from D-Case Editor. The node G 18 in the rightmost branch in Figure 8 appears when a benchmark scenario is imported to D-Case Editor, and the node E 11 appears when the benchmark test completes. The node E 11 in the figure is filled in with blue, which means that the result (a maximum latency of 664.0ms) satisfied the requirement (3000ms). C. Discussion. Figure 8.. Figure 9.. DS-Bench Results as Evidence. A Benchmark Scenario for the Demonstration. programs or anomaly loads on that node. On the timeline, schedules for benchmark programs are shown with blue bars, and the schedule for the anomaly load is shown as a red line. As a benchmark program, an HTTP server measurement tool based on httperf [24] is used. Httperf is modified so that it can measure the downtime of a web server. As the anomaly load, a script shuts down a network link by controlling a network switch via the SNMP protocol. The overall benchmark lasts for 60 seconds, at 30 seconds from the start, the network link for server node 1 is shut down. From that point, server node 1 is no longer accessible, but the service continues because server node 2 handles all requests. In this scenario, the parameters for httperf (the total number of connections and the request rate) are marked as customizable parameters. Also, the result of the benchmark is defined as the maximum latency observed in all request attempts for all client machines, which is obtained using the reduction feature. A stakeholder of the system (say, the operator of the site) conducts the benchmarking test from D-Case Editor. First, the above benchmark scenario is imported from DSBench to D-Case Editor. Then benchmark parameters and the expected result are set (Figure 2). Finally the benchmark. As shown in the above example, currently D-Case Editor conducts only one benchmark run for obtaining one instance of evidence. However this is sometimes not sufficient, because benchmark results may differ for each benchmark run. Therefore, some features, like repeating the same benchmark test several times automatically, then applying some “data reduction” operations (e.g. ave/max/min) to these results to obtain instance of an evidence may be needed. Currently neither D-Case Editor nor DS-Bench have such a feature, but it can be implemented. How many times the benchmark should be repeated, or how the result should be treated (e.g. whether the average works, or the maximum should be taken) to be used as an instance of evidence, are discussed and agreed upon among the stakeholders of the system. In the experiment described above, it was found by a visual inspection of the access logs of the web server that the web server worked properly at the TCP level, however, the Wiki application did not work correctly (it returned HTTP 503 status). In order to validate the dependability of the system thoroughly, the D-Case diagram should have mentioned to the application-level results such as HTTP status of the requests, and the benchmarking tool should have checked the status. V. C ONCLUSION This paper has described DS-Bench Toolset, a toolset for assuring system dependability based on cases and quantitative measurements. The toolset consists of D-Case Editor, DS-Bench, and D-Cloud. D-Case Editor is an assurance case editor which collaborates with DS-Bench and D-Cloud to treat the benchmark test results as evidences for the dependability of the system. DS-Bench is a software framework for conducting several dependability indicator measurements according to a specific scenario which consists of benchmarking programs and anomaly generators. Both benchmark programs and anomaly generators may be existing programs or newly created ones. D-Cloud is a test environment that supports both physical and virtual machines as target machines for a DS-Bench benchmark scenario. D-Cloud also supports several types of fault injection by emulating hardware faults. We have demonstrated an example of a use case of DS-Bench Toolset using a simple web server system, and showed how DS-Bench Toolset has provided evidence of the dependability of the system..

(85) Future work will consist of attempts to apply DS-Bench Toolset to a real system development and evaluation process. In addition, exploiting existing benchmark results stored in the database should be discussed further. For instance, it may be useful to search the database for existing results that may be suitable as evidences for the system currently focused on, allowing users to skip the benchmark execution. ACKNOWLEDGMENT This work has been supported by the JST CREST “Dependable Operating System for Embedded Systems Aiming at Practical Application” project. D-Case Editor has been developed with Fuji Xerox Co., Ltd. Especially, we thank Mr. Hajime Ueno for designing DS-Bench Toolset with us. Windows NT is a registered trademark of Microsoft Corporation in the U.S. and other countries. Eclipse is a trademark of Eclipse Foundation, Inc. Intel and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Other products or brand names appear in this paper may be the trademarks of their owners. Last but not least, we thank the anonymous reviewers for many helpful comments. R EFERENCES [1] A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, “Basic concepts and taxonomy of dependable and secure computing,” IEEE Trans. Dependable Secure Comput., vol. 1, pp. 11–33, 2004. [2] K. Kanoun et al., “DBench dependability benchmarks,” Project Full Final Report, DBench Project, May 2004. [Online]. Available: http://spiderman2.laas.fr/DBench/Final/DBench-complete-report.pdf [3] J. Dur˜aes et al., “Dependability benchmarking of webservers,” in Proc. 23rd Intl. Conf. on Comput. Safety, Reliability and Security (SAFECOMP 2004), ser. LNCS, vol. 3219, 2004, pp. 297–310. [4] J. Dur˜aes and H. Madeira, “Generic faultloads based on software faults for dependability benchmarking,” in Proc. DSN, 2004, pp. 285–294. [5] M.-E. Begin et al., “Build, configuration, integration and testing tools for large software projects: ETICS,” in Proc. Rapid Integration of Softw. Eng. Techniques, ser. LNCS, vol. 4401, Sep. 2007, pp. 81–97. [6] S. Han, K. Shin, and H. Rosenberg, “DOCTOR: an integrated software fault injection environment for distributed real-time systems,” Intl. Comput. Performance and Dependability Symposium, p. 0204, 1995. [7] J. Carreira, H. Madeira, and J. G. Silva, “Xception: A technique for the experimental evaluation of dependability in modern computers,” IEEE Trans. Softw. Eng., vol. 24, pp. 125–136, 1998.. [8] A. Baldini, A. Benso, S. Chiusano, and P. Prinetto, “’BOND’: An interposition agents based fault injector for Windows NT,” in Proc. IEEE Intl. Symposium on Defect and Fault-Tolerance in VLSI Syst. (DFT ’00), 2000, p. 387. [9] S. Potyra, V. Sieh, and M. D. Cin, “Evaluating fault-tolerant system designs using FAUmachine,” in Proc. Workshop on Eng. Fault Tolerant Systems (EFTS ’07), 2007, p. 9. [10] D. Stott et al., “NFTAPE: a framework for assessing dependability in distributed systems with lightweight fault injectors,” in Proc. Comput. Performance and Dependability Symposium (IPDS 2000), 2000, pp. 91–100. [11] Engineering Safety Management, Issue3 (Yellow Book 3), Vol. 1 Fundamentals, and Vol. 2 Guidance, Railtrack, 2000. [12] C. C. Howell et al., Ed., Proc. Workshop on Assurance Case (DSN 2004), 2004. [13] ASCE tools. [Online]. Available: http://www.adelard.com/web/hnav/ASCE/choosingasce/cae.html [14] T. Kelly and R. Weaver, “The goal structuring notation – a safety argument notation,” in Proc. Workshop on Assurance Cases (DSN 2004), Jul. 2004. [15] P. J. Graydon, J. C. Knight, and E. A. Strunk, “Assurance based development of critical systems,” in Proc. DSN, 2007, pp. 347–357. [16] D-Case Editor. [Online]. Available: http://www.il.is.s.utokyo.ac.jp/deos/dcase/ [17] Y. Matsuno and K. Taguchi, “Parameterised argument structure for GSN patterns,” in Proc. 11th Intl. Conf. on Quality Softw. (QSIC 2011), 2011, pp. 96–105. [18] T. Hanawa et al., “Large-scale software testing environment using cloud computing technology for dependable parallel and distributed systems,” in Proc. 2nd Intl. Workshop on Softw. Testing in the Cloud (STITC 2010), Apr. 2010, pp. 428–433. [19] T. Banzai et al., “D-Cloud: Design of a software testing environment for reliable distributed systems using cloud computing technology,” in Proc. 2nd Intl. Symposium on Cloud Computing (Cloud 2010), May 2010, pp. 631–636. [20] T. Hanawa et al., “Customizing virtual machine with fault injector by integrating with SpecC device model for a software testing environment D-Cloud,” in Proc. 16th IEEE Pacific Rim Intl. Symposium on Dependable Computing (PRDC ’10), Dec. 2010, pp. 47–54. [21] OpenStack: The open source, open standards cloud. [Online]. Available: http://openstack.org/ [22] MoinMoin, “The MoinMoin Wiki Engine,” http://moinmo.in/. [23] H. Fujita and Y. Ishikawa, “Anytime available single IP address cluster,” in Proc. 13th IEEE Intl. High Assurance Syst. Eng. Symposium (HASE 2011), Nov 2011, pp. 168–173. [24] D. Mosberger and T. Jin, “httperf: a tool for measuring web server performance,” SIGMETRICS Performance Evaluation Review, vol. 26, no. 3, pp. 31–37, 1998..

(86)