Trust, systems and accidents: Designing complex systems

Darren Dalcher

doi:10.1109/ECBS.2003.1194780

Trust, systems and accidents: Designing complex systems

Darren Dalcher

Hertfordshire Business School

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.

Original language	English
Title of host publication	Procs10TH IEEE Int Conf and Workshop on the Engineering of Computer-Based Systems
Subtitle of host publication	ECBS 2003
Editors	S. Kawada
Place of Publication	LOS ALAMITOS
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	31-38
Number of pages	8
ISBN (Print)	0-7695-1917-2
DOIs	https://doi.org/10.1109/ECBS.2003.1194780
Publication status	Published - 2003
Event	10th IEEE International Conference and Workshop on the Engineering of Computer-Based Systems - HUNTSVILLE, United States Duration: 7 Apr 2003 → 10 Apr 2003

Conference

Conference	10th IEEE International Conference and Workshop on the Engineering of Computer-Based Systems
Country/Territory	United States
City	HUNTSVILLE
Period	7/04/03 → 10/04/03

Access to Document

10.1109/ECBS.2003.1194780

Cite this

@inproceedings{e83dd4d5ea034ea39c02fef197c35f3a,

title = "Trust, systems and accidents: Designing complex systems",

abstract = "Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.",

author = "Darren Dalcher",

year = "2003",

doi = "10.1109/ECBS.2003.1194780",

language = "English",

isbn = "0-7695-1917-2",

pages = "31--38",

editor = "S. Kawada",

booktitle = "Procs10TH IEEE Int Conf and Workshop on the Engineering of Computer-Based Systems",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "10th IEEE International Conference and Workshop on the Engineering of Computer-Based Systems ; Conference date: 07-04-2003 Through 10-04-2003",

}

Dalcher, D 2003, Trust, systems and accidents: Designing complex systems. in S Kawada (ed.), Procs10TH IEEE Int Conf and Workshop on the Engineering of Computer-Based Systems: ECBS 2003. Institute of Electrical and Electronics Engineers (IEEE), LOS ALAMITOS, pp. 31-38, 10th IEEE International Conference and Workshop on the Engineering of Computer-Based Systems, HUNTSVILLE, United States, 7/04/03. https://doi.org/10.1109/ECBS.2003.1194780

TY - GEN

T1 - Trust, systems and accidents

T2 - 10th IEEE International Conference and Workshop on the Engineering of Computer-Based Systems

AU - Dalcher, Darren

PY - 2003

Y1 - 2003

N2 - Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.

AB - Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.

U2 - 10.1109/ECBS.2003.1194780

DO - 10.1109/ECBS.2003.1194780

M3 - Conference contribution

SN - 0-7695-1917-2

SP - 31

EP - 38

BT - Procs10TH IEEE Int Conf and Workshop on the Engineering of Computer-Based Systems

A2 - Kawada, S.

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - LOS ALAMITOS

Y2 - 7 April 2003 through 10 April 2003

ER -

Trust, systems and accidents: Designing complex systems

Abstract

Conference

Access to Document

Fingerprint

Cite this