ANTF Response to the TEF

Response to the BIS Committee about the proposed Teaching Excellence Framework (TEF) on behalf of the UK Association of National Teaching Fellows

By Sally Brown, CANTF Chair

Click to download a PDF of this post –> CANTF TEF response 30 OCTFINAL

This response is submitted on behalf of members of the UK National Teaching Fellows community as represented by the Committee of the Association of National Teaching Fellows: all NTFs have had a chance to view and contribute to this response. We have chosen particularly to respond to questions 3 to 6.

National Teaching Fellowships are awarded to higher education teachers who have been recognised at a national level as excellent university practtioners through a competitive, peer-evaluated process and are therefore uniquely placed to comment on teaching excellence. Each university and all FE colleges that meet certain criteria can submit up to three submissions per year, and the scheme started in 2000. Originally 20, and now a total of 55 university academics and professionals who support HE learning from England, Northern Ireland and Wales are awarded National Teaching Fellowships each year. The National Teaching Fellowship community comprises around 650 members who are represented by the Committee of the Association for National Teaching Fellows.

In July 2015, the Association of National Teaching Fellows organised a TEF consultation event in Oxford, attended by representatives of BIS and HEFCE, and the present response draws on that event as well as on contributions from other National Teaching Fellows.

Summary

National Teaching Fellows are delighted that the government is aiming to recognise the importance of excellent teaching and acknowledge the value of professionalism, giving teaching equal status with research in universities.

We are keen that the TEF should be multi-dimensional, using data already available to universities as far as possible. We suggest that measuring teaching excellence is not a simple or straightforward task, and that the proposal to use existing metrics in the first year of the scheme is likely to result in skewed outcomes. There are widespread and genuine worries that the wrong metrics for a teaching excellence framework will be chosen. We are keen that any measures used to recognise excellent teaching are ones truly representing improvements which add value to the student experience, brought about by interactions between HEI teachers and students, rather than extraneous factors representing existing differences in the level of advantage of the student body in different institutions.

We are very concerned that if the TEF process is expected to serve too many conflicting purposes (for example, both guiding student choices and providing a justification for the increase of above-inflation fees), it is likely to prove unmanageable. We do not welcome any linkage between the TEF and fee levels. NTFs, as the only community of peer-reviewed and nationally-recognised excellent HE teachers would be very well placed to be represented on any group taking the TEF forward.

National Teaching Fellowship applications are prepared annually by HEIs seeking to gain recognition for their excellent teachers, and we propose that these applications could provide a readily-available set of case studies demonstrating impact and pedagogic leadership.

Question 3: What should be the objectives of a Teaching Excellence Framework (‘TEF’)? 

The TEF is a means by which individual and collective teaching excellence is to be recognised in universities and other institutions offering higher education, including Further Education Colleges, (hereafter termed HEIs) giving equal status to teaching to that accorded to research. In recent years there has been an imbalance in esteem, caused largely by HEIs over-focusing on the Research Excellence Framework, thereby giving perverse incentives for HEIs to overlook the importance of teaching.

a. How should a TEF benefit students? Academics? Universities? 

A TEF should benefit prospective students by allowing them to judge the relative teaching excellence of different departments or schools (rather than institutions), as they consider where they want to study. A TEF should benefit existing students, as institutions/departments strive to increase their teaching excellence as measured by the TEF. Furthermore, TEF and other initiatives being advanced by the funding councils can offer greater opportunities for students to engage meaningfully in and thereby influence quality assurance and enhancement activities.

A TEF could benefit academics whose teaching is excellent, giving them greater recognition for their work, and could incentivise HEI teachers who wish to enhance their teaching to place an increased focus on the importance of this aspect of their work. This could thereby counterbalance the current excessive prioritisation of research by some HEIs by emphasising the value of teaching and the scholarship of teaching (Boyer, 2014). A TEF could reinforce the importance of providing sound, base-level training for teaching in higher education, offering HEI teachers better levels of confidence and self-efficacy, as well as helping their managers to prioritise ongoing professional development opportunities. We welcome further moves towards professionalising teaching in higher education. Furthermore, we welcome the opportunity to enhance teaching through more research and evidence-informed approaches, and capturing excellence through pedagogic scholarship and CPD, so that the sector more widely can benefit.

A TEF could benefit Universities and the wider higher education community by providing more information to prospective students and other stakeholders including parents and carers (who frequently financially support them) about the quality of teaching in any institution. HEIs whose overall teaching excellence was found to be high, may then attract more students, and potentially more academically-able ones, to apply to study there. Through high emphasis on teaching excellence, HEIs that make student learning central to their missions could be rewarded for their endeavours, and this in turn could encourage universities to build a system of reward and recognition for teaching excellence.

It is proposed in the consultation that as a result of being highly recognised for teaching excellence, an institution may be allowed to charge higher fees: many NTFs feel this is a very bad idea.

b. What are the institutional behaviours a TEF should drive? How can a system be designed to avoid unintended consequences? 

We consider that institutional behaviours which could be involved in a TEF should include:

1. That the HEI recognises and rewards excellent teaching, e.g. by supporting accreditation through the UK Professional Standards Framework run by the Higher Education Academy (HEA) which offers 4 categories of Fellowship, and by promoting a cadre of promoted staff on the grounds of their excellent teaching (in some HEIs these are termed Teacher Fellows).
2. An indication of the proportion of professors who have achieved this status on the basis of their outstanding teaching, rather than just research.
3. Evidence that students are involved in assuring and enhancing teaching at all stages, for example, through aiding curriculum design and working as partners in curriculum review, and that there are robust systems for training, developing, supporting, valuing and making good use of student representatives.
4. Evidence that all new-to-HE staff are trained and supported through their early years of teaching (linked to probation) including graduate teaching assistants, sessional and fractional staff, and that career-wide continuing professional development is provided for all who teach, with take up of CPD monitored. A metric could include average institutional (or departmental/school) expenditure per head on staff development concerning teaching, learning and assessment, together with average hours used by staff engaging with pedagogic development.
5. Evidence that scholarship and evidence-based practice are valued by the HEI, as indicated for example by the number and quality of peer-reviewed pedagogic publications about teaching, learning and assessment produced, including action-research and teaching-related consultancies.
6. Data that show the number of staff at an HEI who have achieved National Teaching Fellowships over the 15 years the scheme has been running, and how the HEI engages them in enhancing the management of change in learning and teaching practices within and beyond their own institution.
7. Evidence of quality assurance measures meeting expectations, which result in Quality Assurance Agency, and Professional, Statutory and Regulatory Body confidence.
8. Data on the spend per full-time equivalent student on learning environments, libraries and virtual resources for students. Such metrics must be mode-neutral between full-time and part time students, and should recognise that many programmes are managed through distance learning.
9. Evidence that international students are selected appropriately and supported throughout their studies.
10. Evidence that students are satisfied with their learning experiences as indicated by a basket of measures, one of which could be NSS outcomes, although many acknowledge that measuring satisfaction and measuring excellence are very different processes.
11. Evidence that outcomes for students are excellent as indicated by retention data at each level of undergraduate degree programmes, and on Masters programmes, and for PhD completions within reasonable periods.
12. Data demonstrating that students are employed in graduate professions, or are creating successful portfolio/freelance careers in equivalent vocational professions within five years of graduation.
13. Evidence that assessment systems have robust moderation in place to assure standards, with effective use of benchmarks and positive external examiners’ reports.
14. Evidence that shows that steps are taken to advise students on what comprises plagiarism and poor academic conduct, and that poor practice is recorded, acted upon and monitored against national levels.
15. Evidence that HEIs demonstrate investment in student engagement, welfare and pastoral care through student recruitment policies, and throughout the student lifecycle, showing a commitment to inclusivity and redressing all kinds of disadvantage, particularly in terms of demonstrating and monitoring Fair Access and Widening Participation activities.
16. Inclusion of reviewed case studies demonstrating impact on the way that the discipline is taught in the sector. Case studies in relation to individual excellent teaching already exist in the three submissions that HEIs canmake each year for National Teaching Fellowships. Similarly case studies for sustained and effective educational leadership and strategic influence exist already in the successful Principal Fellowship applications that are accredited by the HEA.

Some caveats

Indicators such as first destination of students on graduation can be valuable but are notoriously hard to collect from students who choose not to keep in touch with their universities, and such indicators are therefore innately inaccurate. Furthermore, destination data tends to be aligned to local economic factors and students’ socio-economic backgrounds.

Indicators such as the proportion of students who get ‘good’ degrees i.e. Firsts and upper seconds are readily gameable indicators, and it is acknowledged that good degrees mean different things in different universities.

Similarly high retention figures could be an indicator of good teaching and learning support but also could be more representative of universities’ intakes, so wider contextual information is necessary, including potentially divergence from benchmarks. Students who enter universities with disadvantages have higher drop-out rates (Yorke, 2004), so caution must be exercised to avoid channelling universities into risk averse behaviours, by choosing only students from privileged backgrounds more likely to stay the course. There is ample evidence that retention and achievement rates not only vary between subjects but also reflect patterns of social and economic inequality.

We welcome universities being required to demonstrate widening participation, but this is not an indication of teaching excellence. Measures of learning gain for students from disadvantaged backgrounds achieving degrees and other HE awards can be a good indicator of excllent teaching but other factors are likely to be involved. As the Prime Minister acknowledged in his 2015 Conservative Party conference speech, inequalities and prejudices impact on student recruitment processes and these in turn impact upon graduate employment outcomes.

The avoidance of unintended consequences

We suggest that measuring teaching excellence is not a simple or straightforward task, and that the proposal to use only existing metrics in the first year of the scheme is likely to result in skewed outcomes.

Outcomes of the National Student Survey may be a useful indicator at a subject level, but they measure student satisfaction rather than student engagement or learning gain, and therefore are only a poor proxy for teaching excellence. NSS results tend to vary by discipline, with students consistently reporting higher levels of satisfaction with some subjects than others.

If indicators such as NSS, first destination employment data and retention data are to be key elements of the TEF, each indicator must be benchmarked to take account of student characteristics, discipline-level differences and other factors such as the region in which a university is based. This could level the playing field between institutions with different profiles and missions in a way that existing league tables fail to achieve. Additionally, the indicators that are available can be improved: revisions to the National Student Survey for 2017 are expected to include new questions that aim to measure student engagement. HEFCE is also undertaking 12 pilot projects to explore how learning gain might be measured. The provisions of the Small Business, Enterprise and Employment Act, passed earlier this year, should also provide more accurate data on graduate earning and employment.

Among measures we would not find useful, as they represent existing advantage among students on entry, are metrics which take into account salaries on graduation, as they implicitly advantage certain subject areas and disciplines as well as enforcing differences that are implicit at entry. For example some high paying blue-chip employers only recruit from a very limited pool of universities. Some disciplines like art and design rarely produce graduates with immediate high incomes, whereas others such as banking, finance and accounting are more likely to lead to initial high salaries.

Staff student ratios tend to be related to funding of universities, and therefore are not necessarily an indication of good teaching but could provide contributory evidence of the importance of afforded to it.

c. How should the effectiveness of the TEF be judged?

The effectiveness of the TEF could be judged by how well the measures of teaching excellence listed above are reflected. Any system adopted must be multi-dimensional but should not be resource-intensive and must be seen to be fair and equitable by the sector, if judicial review and expensive complaints processes are to be avoided.

Representatives of the National Teaching Fellowship community will have confidence that the TEF is trustworthy if outcomes are based on evidence rather than historic reputations, and if the means by which judgments of teaching excellence are made are transparent and demonstrably fair.

Since there are significant variations between the performances of different departments within any university, we would propose that measures of teaching excellence should be at a subject rather than an institutional level, recognising the importance of subject differences and signature pedagogies (Shulman, 2005) although this might be problematic if the TEF is to be linked to the increasing of fees for excellent teaching.

There is a range of ways in which these two alternative approaches (Institutional level and Subject level) could be combined. One option would be to require the institution as a whole to reach a certain threshold level of performance and then allow individual units to be entered into an assessment process. Such an approach has been compared to the process by which the Athena Swan gender equality award system works, with potentially a similar gold, silver and bronze departmental rating system for teaching excellence. This approach would allow for the recognition of achievements at the level at which academic staff and students tend to sit. It would also allow for a clear comparison between courses in the same subject at different universities. However, if the link to fees remains at the institutional level, problems aggregating departmental ratings would always remain.

Question 4. How should the proposed TEF and new quality assurance regime fit together? 

Representatives of the National Teaching Fellowship community argue that the TEF and any new quality assurance regime must be manageable, in that data prepared and presented by HEIs should only be required to be presented once. This means both systems must articulate fully, and arguably form part of a single cohesive system, whilt acknowledging the distinctions between quality assurance and teaching excellence. We would not wish to see an approach that mirrors former QAA subject review, which was arduous, time consuming and subject to gaming. Nor would we wish to see over-simplistic approaches that use only currently publicly available data and which therefore could not accurately reflect the dimensions of excellence for which we argue.

Representatives of the National Teaching Fellowship community have welcomed the support of HEFCE, DENI and HEFCW over the 15 years of our existence, and consider that HEFCE and its agencies would be well placed to offer a coherent approach to the administration of the TEF and QA regime.

Question 5. What do you think will be the main challenges in implementing a TEF? 

Representatives of the National Teaching Fellows community in our discussions agreed that deciding on the most appropriate metrics for teaching excellence is very challenging indeed.

Appropriately balancing the different elements of data used to reflect the teaching excellence of a whole institution seem to us impractical and we would therefore argue for a subject-based approach. Alternatively, working out how best to establish teaching excellence in particular disciplines, rather than whole institutions, where individual departments/schools are measured in terms of teaching excellence could be viable.

Question 6. How should the proposed connection between fee level and teaching quality be managed?

a. What should be the relationship between the TEF and fee level?

Representatives of the National Teaching Fellowship community feel it is extremely hazardous to link increases in fees above inflation to TEF scores until the process has been piloted and stress-tested, and hence we argue this should not be done in the first phase of TEF. Only when an approach is arrived at which is demonstrably fair should a linkage with higher fees be considered if at all.

b. What are the benefits or risks of this approach to setting fees?

Benefits of linking teaching excellence to increased fees might provide some students with clearer choices when selecting courses and universities. Additionally, as discussed above, the knock-on effects of such a linkage may drive a range of desired behaviours within institutions.

Risks of linking teaching excellence to increased fees include:

• Potential failure to have confidence in the system as fair, thereby leaving universities open to legal challenges and judicial review;
• Problems concerning different fees being set for different subjects in the same HEI, and the complexities of administering them;
• Potential risks that such a regime might be perceived as allowing universities offering excellent teaching being only available to students who can afford them, consigning other students to what might be seen as ‘second best’. Only generous and accessible bursaries for students from disadvantaged backgrounds could redress this problem;
• Risk that international students, who are currently recruited by most HEIs, considering that only those universities charging high fees are worth applying to;
• Further reinforcement and hardening of existing prejudices about the relative qualities of different HEIs and different subject areas which are currently masked by equivalent fee regimes.

  

References

Boyer, E. L. (2014) Scholarship reconsidered: Priorities of the professoriate.

Shulman, L. (2005) The signature pedagogies of the professions of law, medicine, engineering, and the clergy: Potential lessons for the education of teachers. In Talk Delivered at the Math Science Partnerships (MSP) Workshop: “Teacher Education for Effective Teaching and Learning” Hosted by the National Research Council’s Center for Education, February (pp. 6-8).

Yorke, M. (2004) Leaving early: Undergraduate non-completion in higher education, London: Routledge.