The problems of assessing competence
It was interesting to read Simon Reddy’s article in FE News, The Problem with Further Education and Apprenticeship Qualifications, lamenting the low standard of training in plumbing the UK and the problems with the assessment of National Vocational Qualifications.
Simon reported from his research saying:
There were structural pressures on tutors to meet externally-imposed targets and, judging from the majority of tutors’ responses, the credibility of the assessment process was highly questionable.
Indeed, teachers across the three college sites in my study were equally sceptical about the quality of practical plumbing assessments.
Tutors in the study were unanimous in their judgements about college-based training and assessments failing to adequately represent the reality, problems and experiences of plumbers operating in the workplace.
In order to assess the deviation away from the original NVQ rules, he said, “it is important to understand the work of Gilbert Jessup, who was the Architect of UK competence-based qualifications.
Jessup (1991: 27) emphasised ‘the need for work experience to be a valid component of most training which leads to occupational competence’. Moreover, he asserted that occupational competence ‘leads to increased demands for demonstrations of competence in the workplace in order to collect valid evidence for assessment’.
As a representative of the Wesh Joint Education Committee, I worked closely with Gilbert Jessop in the early days of NVQs. Much (probably too much) of our time was taken with debates on the nature of competence and how assessment could be organised. I even wrote several papers about it – sadly in the pre digital age.
But I dug out some of that debate in a paper I wrote with Jenny Hughes for the European ICOVET project which as looking at the accreditation of informal learning. In the paper – with the snappy title ‘The role and importance of informal competences in the process of acquisition and transfer of work skills. Validation of competencies – a review of reference models in the light of youth research: United Kingdom.’
In the introduction we explained the background:
Firstly, in contrast to most countries in continental Europe, the UK has long had a competence based education and training system. The competence based National Vocational Qualifications were introduced in the late 1980s in an attempt to reform and rationalise the myriad of different vocational qualifications on offer. NVQs were seen as separate from delivery systems – from courses and routes to attain competence. Accreditation regulations focused on sufficiency and validity of evidence. From the very early days of the NVQ system, accreditation of prior learning and achievement has been recognised as a legitimate route towards recognition of competence, although implementation of APL programmes has been more problematic. Thus, there are few formal barriers to access to assessment and accreditation of competences. That is not to say the process is unproblematic and this paper will explore some of the issues which have arisen through the implementation of competence based qualifications.
We went on to look at the issue of assessment:
The NVQ framework was based on the notion of occupational competence. The concept of competence has been a prominent, organising principle of the reformed system, but has been much criticised (see, for example, Raggatt & Williams 1999). The competence-based approach replaced the traditional vocational training that was based on the time served on skill formation to the required standard (such as apprenticeships). However, devising a satisfactory method of assessing occupational competence proved to be a contentious and challenging task.
Adults in employment who are seeking to gain an NVQ will need a trained and appointed NVQ assessor. Assessors are appointed by an approved Assessment Centre, and can be in-house employees or external. The assessor will usually help the candidate to identify their current competences, agree on the NVQ level they are aiming for, analyse what they need to learn, and choose activities which will allow them to learn what they need. The activities may include taking a course, or changing their work in some way in order to gain the required evidence of competence. The opportunity to participate in open or distance learning while continuing to work is also an option.
Assessment is normally through on-the-job observation and questioning. Candidates must have evidence of competence in the workplace to meet the NVQ standards, which can include the Accreditation of Prior Learning (APL). Assessors will test the candidates’ underpinning knowledge, understanding and work-based performance. The system is now intended to be flexible, enabling new ways of learning to be used immediately without having to take courses.
The system is characterised by modular-based components and criterion-referenced assessment. Bjornavald also argues that the NVQ framework is output-oriented and performance-based.
We outlined criticisms of the NVQ assessment process
The NCVQ methods of assessing competence within the workplace were criticised for being too narrow and job-specific (Raggatt & Williams 1999). The initial NVQs were also derided for applying ‘task analysis’ methods of assessment that relied on observation of specific, job-related task performance. Critics of NVQs argued that assessment should not just focus on the specific skills that employers need, but should also encompass knowledge and understanding, and be more broadly based and flexible. As Bjornavald argues, ‘the UK experiences identify some of these difficulties balancing between too general and too specific descriptions and definitions of competence’. The NVQs were also widely perceived to be inferior qualifications within the ‘triple-track’ system, particularly in relation to academic qualifications (Wolf 1995; Raffe et al 2001; Raggatt 1999).
The initial problems with the NVQ framework were exacerbated by the lack of regulatory powers the NCVQ held (Evans, 2001). The system was criticized early on for inadequate accountability and supervision in implementation (Williams 1999), as well as appearing complex and poorly structured (Raffe et al 2001).
We later looked at systems for the Accreditation of Prior Learning (APL).
Currently the system relies heavily on the following basic assumptions: legitimacy is to be assured through the assumed match between the national vocational standards and competences gained at work. The involvement of industry in defining and setting up standards has been a crucial part of this struggle for acceptance, Validity is supposed to be assured through the linking and location of both training and assessment, to the workplace. The intention is to strengthen the authenticity of both processes, avoiding simulated training and assessment situations where validity is threatened. Reliability is assured through detailed specifications of each single qualification (and module). Together with extensive training of the assessors, this is supposed to secure the consistency of assessments and eventually lead to an acceptable level of reliability.
A number of observers have argued that these assumptions are difficult to defend. When it comes to legitimacy, it is true that employers are represented in the above-mentioned leading bodies and standards councils, but several weaknesses of both a practical and fundamental character have appeared. Firstly, there are limits to what a relatively small group of employer representatives can contribute, often on the basis of scarce resources and limited time. Secondly, the more powerful and more technically knowledgeable organisations usually represent large companies with good training records and wield the greatest influence. Smaller, less influential organisations obtain less relevant results. Thirdly, disagreements in committees, irrespective of who is represented, are more easily resolved by inclusion than exclusion, inflating the scope of the qualifications. Generally speaking, there is a conflict of interest built into the national standards between the commitment to describe competences valid on a universal level and the commitment to create as specific and precise standards as possible. As to the questions of validity and reliability, our discussion touches upon drawing up the boundaries of the domain to be assessed and tested. High quality assessments depend on the existence of clear competence domains; validity and reliability depend on clear-cut definitions, domain-boundaries, domain-content and ways whereby this content can be expressed.
It’s a long time since I have looked at the evolution of National Vocational Qualifications and the issues of assessment. My guess is that the original focus on the validity of assessment was too difficult to implementing practice, especially given the number of competences. And the distinction between assessing competence and assessing underpinning knowledge was also problematic. Easier to move to multiple choice computerized testing, administered through colleges. If there was a need to assess practical competences, then once more it would be much simpler to assess this in a ‘simulated’ workshop environment than the original idea that competence would be assessed in the real workplace. At the same time the system was too complicated. Instead of trusting workplace trainers to know whether an apprentice was competent, assessors were themselves required to follow a (competence based) assessors course. That was never going to work in the real world and neither was visiting external assessors going to deliver the validity Gilbert Jessop dreamed of.
If anyone would like a copy the paper this comes from just email me (or add a request in the comments below). Meanwhile I am going to try to find another paper I wrote with Jenny Hughes, looking at some of the more theoretical issues around assessment.
