Website evaluation 2010
Progress report 2010
The NHS medical careers website, which is
managed by Health Education KSS on behalf of the Department of
Health, was first launched in November 2008. Since then its user
base has grown from less than 4000 visits per month in July 2009 to
over 27000 visits in October 2011.
It is a year since an external evaluation of
the website was carried out by Robin Mellors-Bourne from the
Careers Research Advisory Centre (CRAC) in November 2010. The
evaluation looked principally at the awareness of the site amongst
its target audience of medical students and medical trainees, why
users came to the site, their overall impressions and how it has
helped with career decision-making.
We thought it might be of interest to feedback
some of the results of the evaluation as they give some insight
into medical students’ and foundation trainees’ awareness and
thoughts about their careers.
The evaluation found that around 60% of
foundation trainees are aware of the site as well as nearly half of
all medical students (see Box 1 for further details).
Box
1: Awareness of the site
|
%
|
Overall
|
Medical students
|
FY 1 trainees
|
FY 2 trainees
|
Specialty trainees
|
|
Aware of site
|
49
|
48
|
57
|
67
|
39
|
|
Used site
|
42
|
43
|
44
|
59
|
31
|
|
Of users:
|
|
|
|
|
|
|
Once
|
28
|
30
|
31
|
23
|
25
|
|
2-3 times
|
41
|
44
|
36
|
44
|
40
|
|
> 3 times
|
31
|
26
|
33
|
34
|
34
|
CRAC undertook the research over the
summer/autumn of 2010 with an online survey which was backed up by
telephone interviews with selected participants. Two e-mail
campaigns were undertaken, one was to current registered users of
the site (all users have the option of registering if they want
access to the free interactive tools on the site). The other email
campaign was sent via medical school and foundation school contacts
in order to canvas views more widely. In total 1459 usable
responses were received from students and trainees in almost every
UK medical school and postgraduate deanery. A summary of the
findings of the independent evaluation are represented below:
User impressions of the site
It was found that the reasons for accessing
the site differed slightly for the different user groups. Greater
awareness of career options was the main reason for both medical
students and foundation trainees and this was particularly so for
medical students. It is not surprising therefore that the specialty
pages and the careers options pages are viewed as the most useful
sections on the website as can be seen in Box 2 below:
Box
2: Users impressions of content on the
site 
* Rated by respective
target audience groups only.
The external evaluation also looked at the use
of the media-rich content on the site. The NHS medical careers team
have produced six short video-casts to date and the site also links
to other media-rich content filmed by other sources. In addition
the website features some interactive tools which aid
self-awareness and career decision-making. The video content
and interactive online tools usage figures are shown in Box 3:
Box
3: Usage and users impressions of video
content and the interactive online tools
User impressions of the Video content (V) and Interactive Tools
(T)
The views of non-users were also examined in
the evaluation and over three-quarters said that now they are aware
of the site they expected to use it. There were very few that
suggested that they did not have any need for such careers
provision. Over half of the non-users expected to use the
interactive career tools and over a quarter the video content about
specialties.
Measures of impact
The independent evaluation of the website
sought to look at what users did next after viewing the website. It
was found that almost 50% reflected on the information/learning and
over 40% visited other career websites to obtain more online
information. Over 30% of Foundation Year 2 and Specialty trainees
stated that it prompted them to make decisions about applications.
Box 4 below demonstrates the overall value of the website:
Box
4: Overall value ~ User-perceived
impacts
|
%
(Expressed as % Strongly agree or agree).
|
All users
|
Frequent users
|
Medical students
|
FY 1 trainees
|
FY 2 trainees
|
Specialty trainees
|
|
Influenced specialty decision
|
59
|
70
|
62
|
52
|
64
|
54
|
|
Influenced Foundation choice
|
45
|
61
|
47
|
38
|
37
|
-
|
|
Led to adviser conversation
|
47
|
59
|
36
|
49
|
52
|
68
|
|
Recommended site to others
|
63
|
75
|
62
|
57
|
64
|
65
|
The independent evaluation also compared the
impact on frequent users v non-users and found that:
- A higher proportion of medical students who were frequent users
had a good idea of where to undertake foundation training (77% v.
47% of non-users)
- Higher proportions of frequent users at all stages were fairly
sure they know which specialty they would pursue, looking at for
example Foundation Year 2 trainees (95% v. 72% of non-users).
Within the survey respondents were invited to
make additional comments and over 100 responses were gathered of
which many expressed positive statements about how they found the
site useful. The independent report highlights some of these
comments, this is shown in Box 5:
|
Box
5: Comments from users about the
site
‘It’s helpful to have a reputable resource
for careers information all in one place.’
‘I feel it’s a really good website but I
had never heard of it before until stumbling across it on Google.
If it was highlighted to everyone at medical school I think it
would be a great resource for all of us.’
‘I think it’s a great website with very
useful, relevant and up-to-date information which has been great
for myself in deciding which areas of medicine I would be more
suited to.’
‘It is an invaluable tool for foundation
doctors considering their specialist training career.’
‘It’s an extremely valuable website I was
delighted to find it.’
‘It is a fantastic resource, and a real
move forward from where we were previously (a lot of disparate
information from a range of sources that was often difficult to
find or navigate).’
‘The site fills a gap in information
provision as the NHS site tended to give a broad view of Medical
Careers. I think it has been a ‘’godsend’’ to me as I seek to
up-to-date and relevant guidance.’
|
Future plans for the website
KSS Deanery is at present re-tendering for the
continuation of the website and has put together information for a
business case which is being progressed by the Department of
Health. We are continuing in our efforts to promote the website to
both medical students and to postgraduate trainees, as well as to
educational supervisors and influential bodies who have a role in
signposting the website to users.
In terms of ensuring the quality of the
website we ask the Royal Colleges to review the specialty pages
once a year and broken links are checked quarterly. We also hold a
focus group of key users who advise us on the content and on
maintaining the impartiality of the website. In addition we are
adding further case-studies to add to the ‘human’ element of the
website.
We are also planning to produce more
media-rich content. Two videos have recently been produced, one
that helps to demystify labour market information and competition
ratios and another introducing the site and highlighting its key
features to new users. We have also produced two audio-casts on the
competition into surgery, one for the undecided and one for those
who are sure they want to apply. We have plans to produce
several further podcasts, in the near future.
One element the evaluation fed back on was
that some users were not identifying between NHS medical careers
and NHS Careers and so we have developed a new graphic in order to
enable users to identify and recall the site more readily.
The full evaluation report of the website can
be downloaded from the following URL: http://kssdeanery.org/document/nhs-medical-careers-report-2010
Lisa Stone – Careers Adviser, KSS Deanery –
November 2011
Correspondance to careers@kssdeanery.ac.uk