home button (image)
Nuclear header image

Nuclear engineering for safety, control and security

13-14 March 2019 | Bristol Marriott Royal Hotel, Bristol

CPD 14 hours

Statistical testing methods for ‘Smart Device’ justification

  • Silke Kuball, Lee Walker: EDF Energy Nuclear Generation
  • Rob Stockham, Sam Faulkner: Moore Industries-Europe, Inc.
  • Benedikt Heinz, Benoit Jouan: Gantner Instruments GmbH

What are smart devices and how are they used in the nuclear industry?

The civil nuclear industry increasingly uses Commercial off The Shelf software-based devices that

  1. have a specific limited functionality (such as transmit a pressure, alarm on high temperature etc),
  2. contain a microprocessor running software to carry out that function,
  3. are pre-developed for a wide range of different customers and industries (not just safety-critical) and
  4. can be configured but not programmed by the end user.

In the UK nuclear industry, we refer to such devices as “smart devices”, which has nothing to do with artificial intelligence but merely refers to the in-built “intelligence” in the form of say firmware. It refers to the fact that the behaviour of these devices cannot be exhaustively tested, the number of inputs combined with internal states is too high and even after many operational use hours one can still encounter a situation where for a specific input/configuration combination, the device exhibits “unexpected behaviour”.

“Smart devices” often carry out what looks like a “simple” function, yet their internal complexity can be substantial. Hence as with all complex designs there is a risk of latent errors and when such devices are used in safety-related or safety-critical contexts, the customer needs to satisfy themselves (and their regulator) that the methods used to reduce the risk of such errors leading to hazardous failures are appropriate, effective and satisfactorily applied. This is usually the aim of using standards and auditing against standards.

How are they usually justified? What are some of the problems with these methods of justifying smart devices?

Where they are used in safety significant contexts, “safety-claims” are attached to devices and these are linked to class, to SILs and to probabilities of failure per demand or per annum. Usually, the UK nuclear industry is employing the EMPHASIS ‘Production Excellence’ approach to perform independent assessments of the development lifecycle of a device. This approach is largely based on IEC 61508. Where there is a lack of compliance with 61508 or an equivalent standard or where there is a lack of ability of the manufacturer to make resource available to support such an assessment, the confidence to be gained from a standards-based approach is limited. In this case, additional measures for device substantiation need to be employed. Also, when the safety claim is quite onerous, e.g. 10-3 pfd or 10-4 pfd, the licensee will carry out additional in-depth independent confidence building. In both these cases, testing of the product with actual use-cases is a technique of great benefit. It allows to observe the behaviour of the product in operation and – if done right – this can significantly add to confidence in the product.

What is statistical device testing and why is it useful for device justification?

Statistical testing is a technique that involves tests built from meaningful “use cases”, suitably randomised, and these tests – if they fulfil certain key conditions - allow to derive an estimate of the in-use failure probability based on the test results. This estimate is in the form of an upper confidence bound on the device probability of failure per demand. Apart from formal proofs, this is the only technique that allows quantification of the dependability of a device. Apart from this, the test cases emanating from the technique are meaningful in that they are steered by expected behaviour of the environment in which the device is to “live”, and this is an effective way of testing.

Part 1 of this workshop will be led by EDF Energy and will explain the fundamental principles of statistical testing as well as show examples of how it has been applied in practice in the UK nuclear industry.

What may the future of smart device justification look like? Will there be demand for engineers to use statistical testing methods?

Standards-based assessments are valuable, but they are not always possible, not always consistent, qualitative rather than quantitative, and do not directly relate to the application context. It seems prudent and pragmatic to complement these with techniques that can reveal potential issues of the product in actual use, such as statistical device testing (ST).

However, there has been resistance against employing ST which -given its effectiveness as a confidence establishing technique- is not a sensible position.

EDF Energy have been involved in a variety of statistical testing projects, which have proven to be beneficial. Some recent ones have been supported by Moore Industries and Gantner Instruments. From this experience the main reasons for the technique not being used more widely seem to be:

  1. the high resource investment in building an automated test-rig and
  2. the perceived complexity of building statistically meaningful test-cases.

For these reasons, and because we believe that a wider more mainstream use of statistical testing would benefit not only the nuclear industry but other industries as well, we have embarked on the task of developing a generic test rig for statistical testing of smart devices. Although this test rig is primarily for smart devices, we believe that some of the general concepts of this test rig can be adapted to other situations such as testing bespoke systems, maybe even FPGA’s or PLC platforms.

Part 2 of this workshop will be led by Moore Industries and Gantner Instruments and they will discuss the implementation of a generic test rig, the main components and their features, status of the test rig development and lessons learned so far.

Book now to learn more about statistical testing methods for ‘Smart Device’ justification

Headline sponsor

Sponsors / exhibitors


Conference registration

Member - £585
Non-member - £685
Student - £150

All prices are per person and are subject to VAT at 20%.