Conducting proficiency tests

Guideline 5.0

(resolution of the General Meeting 2012 in Königswinter)

modified by Guideline 5.1

(resolution passed at the General Meeting in 2019 in Nürnberg)

1     General

1.1  Objective

The following regulations will serve as guideline for the uniform handling of the planning, preparation, execution, evaluation and reporting on proficiency tests within the GFS.

1.2  Scope

This guideline applies to GFS members and to external third parties who have been commissioned with the organisation of proficiency tests.

1.3  The foundations or starting points are described in:

-     'Guidance on the Conduct of Proficiency Tests and Collaborative Exercises within ENFSI': www.enfsi.eu / Standing Committees / QCC

-     ILAC-G22:2004 'Use of Proficiency Testing as a Tool for Accreditation in Testing': www.ilac.org / Publications and Resources / ILAC Guidance Documents (G-Series)

-     ILAC-G13:08/2007 'Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes': www.ilac.org / Publications and Resources / ILAC Guidance Documents (G-Series)

-     ILAC-P9:06/2014 'Policy for Participation in Proficiency Testing Activities': www.ilac.org / Publications and Resources / ILAC Policy Documents (P-Series)

-     71 SD 0 010 'Einbeziehung von Eignungsprüfungen in die Akkreditierung': www.dakks.de / Dokumente / Allgemeine Dokumente

-    ISO/IEC 17043-2010 'Conformity Assessment – General Requirements for Proficiency Testing'

2     Planning

2.1  Anonymization

The anonymization of the participants must be ensured. Only unanimous consent by the participants will permit the revocation of this anonymity. Numbers will be assigned to the individual participants, and their answers will be returned through an appointed representative.

According to the modification, the codes of the possible participants are assigned by a person of trust (Executive Board of the GFS) and thereafter retained. Another person of trust (chairman of the Arbitration and Disciplinary Committee) continuously summarizes the results of the proficiency tests carried out for the individual coding numbers, but receives no knowledge of the assigned names.

2.2  Types of qualification tests

Qualification tests (proficiency tests) may include qualitative identifications (comparisons), quantitative measurements and/or interpretations.

There are two types:

In the first type, the test material (e. g. original document) is to be passed on to the next participant in line with the schedule provided. Also to be observed:

-    identical conditions must be assured, so that all participants receive identical test materials (apart from changes or damage that may result from improper handling),

-    only non-destructive test methods are permitted,

-    sufficient time must be foreseen for passing on the test material within the group of participants (provide schedule).

A second type involves providing each participant with a set of test materials. To be observed here:

-    differences in the individual sets must be taken into consideration in the test evaluation,

-    destructive examinations are possible.

A combination of the two test types, with one set of test materials per group of participants, is also possible.

2.3  Expected result

Before the proficiency test is released, a ‘competent body’ (e. g. Examination Board) should examine it with regard to its conclusiveness and its expected results. The determination of the ‘competent body’ can be communicated to the organiser.

3     Preparation

3.1  Test area/scope

Proficiency tests should correspond to day-to-day casework and should be based on available test methods and tools, as described in Guideline 1.01. Depending on the content of the test, a brief description of the case is possible. Complex details regarding the case, as well as those which might be misleading should be avoided. It should not take more than two working days to complete the test.

3.2  Test material

To enable the comparison of the individual results (evaluation, representativity), all of the participants are to be provided with identical or uniform test material to the extent possible (this applies to tests using various sets of test material). The test material must be clearly marked (incriminating material, comparative material). The provenance of the comparative material (unbiased or ad hoc) should be named.

Preparation of the test material should be documented to make it possible to trace its condition of origin.

4     Execution

4.1  Notification

Participation is obligatory for ordinary members. For candidate members, participation is voluntary.

Notification of the test will be made in advance. This notification will contain the following information:

-    a short description of the proficiency test and its purpose,

-    criteria for participants who are not also members of the GFS,

-    scheduling: Registration deadline, filing deadline,

-    possible costs and payment terms for non-members,

-    name/contact address of the test organiser.

4.2  Packaging/Shipping

The packaging must ensure that the test material will not be damaged or altered. For proficiency tests in which the test material is forwarded to the next participant, it must be ensured that this requirement can be fulfilled by all persons involved.

In this case, it must be very clear to whom the test material is to be forwarded by each individual participant. The organiser is to be informed when the material is forwarded, so that the schedule can be monitored. 

4.3  Answer sheet

The results are to be recorded by the participants on an answer sheet that is provided and that is clearly structured and easy to understand. The test is to be based on a fair, practice-related approach.

Multiple choice questions will simplify the evaluation (uniform assessment scale), but are not appropriate for every case.

The participants are to indicate which test methods and equipment are used.

To ensure a meaningful statistical evaluation of the test, it is permissible to request additional information about the participants (training, experience in the respective field of expertise, expert status, quality control, etc.), as long as their anonymity is ensured.

It must be possible to return the result in anonymised form for the evaluation.

5     Evaluation/Reporting

5.1  Results

The correct test result is not to be communicated by the organiser before the closing date for the returning of the test results.

If this is statistically meaningful and possible on the basis of the test arrangement, the results are to be evaluated and documented, including the statistical methods, standard deviation, and measurement uncertainty. This is the only way to guarantee a uniform and objective evaluation of the test results.

The evaluation results are to be compiled in a suitable form (e. g. chart, table) and presented at one of the following General Meetings. The overall results will be summarised in anonymised form. Individual results will be presented so that each of the participants will be able to identify him/herself, but will not be identified by others. The organiser will comment on the results.

Due to the constancy of the coding, distributions can be determined over the years, but without reference to individual persons. The individual participants can aggregate their personal results independently. This measure shall allow general conclusions about the analysis of the distribution of errors. The person of trust (chairman of the Arbitration and Disciplinary Committee) receives a list of results with coding and dichotomized results (correct / incorrect) from each proficiency test.

5.2  Measures

The result of a proficiency test is to be discussed in the appropriate setting. If this should be necessary (e. g. in the case of a false expertise result, but also failed test preparation or erroneous/insufficient evaluation), appropriate measures should be recommended for all participants (e. g. optimisation of the test methods, changes in training, help in the preparation and evaluation of a proficiency test by scientifically experienced experts). If an expertise is incorrect, assuming the test has been correctly prepared and implemented, responsibility for measures to be taken is with the test participants. The test organiser does not have the required competence. However, he does bear responsibility for the scientifically correct development, implementation and evaluation of the test.

5.3  Keeping of Records