Human Performance Technology (HPT) in surgical education - my experience
In a training
program to teach orthopaedic surgery residents on the microsurgery skills of
repairing the nerves of the hand, the current success rate is not as
expected and the resources that are used to run this workshop is significant.
The problem is,
How to increase
the rate of successful acquisition of the skill of nerve repair in this high
fidelity skill workshop.
Benefits that can be gained by solving the problem
If the success
rate can be increased, the number of trained personnel will be higher
the number of successful nerve repair done in the hospital will be higher
and therefore reducing the number re operations and secondary reconstructive
procedures with better outcomes for the patients.(Mandal & Banerjee, 2012)
What is expected when the problem is solved or
performance is improved?
If the success
rate of skill acquisition in the workshop can be increased the following can be
expected:-
1. number of surgeon competent to perform nerve surgery will be higher
2. the number of successful nerve repair done in the hospital will be
higher
3. reduction in the number re operations for nerve surgery
4. secondary reconstructive procedures for nerve injuries will be less
5. Better outcomes for patients with nerve injury.
Determination of the performance gap:
The learning outcome for this highly targeted workshop was the
ability by the candidates to perform microsurgical repair of a nerve in a
biological model. The assessment of the outcome had been developed with the use
of an objective structured rubrics that had been created. (Appendix 1)
This workshop runs for two days with 10 candidates and approximately
4 faculty members. Estimated cost for the workshop is approximately USD $50,000
per day. Thus poor outcomes for the workshop makes it cost ineffective to
continue to run the workshop.
Assessments were provided as a formative type during the workshop
with immediate feedback so as the candidates were able to learn effectively
with this formative assessment. This was provided for by the faculty and thus
the higher faculty candidate ratio. Following repeated and deliberate practice
of the procedure on the model, they were then assessed summatively which was
recorded digitally and the time taken to perform the task was also recorded.
The final assessment of this exercise was performed by three independent
assessors who were senior surgeons using the rubrics in appendix 1. The rubrics
were developed according to standard protocol used in objective structured
clinical evaluation. As can be seen in the rubrics the descriptors for the
various levels of performance were clearly stated.
From the two workshops that we had conducted the number of
candidates who had a score of four upon five on the Likert score for the
quality of repair was 20%. This was the actual performance of candidates in
this resource intensive workshop that we were conducting. We had produce
measurement tools that were structured and had internal validity to conduct an
assessment of the quality of repair. In the ideal world we would have expected
100% of the candidates being able to perform a good repair according to our
rubrics. Thus currently this workshop was producing a performance gap of 80%.
Measurements of improvement in performance of the candidates will be
conducted using the same rubrics and therefore providing reliability between
the two measurements.
Factors affecting performance:
Areas that will affect performance in this workshop can be divided
into
·
environmental factors and
·
individual factors
For environmental factors that can be a cause for performance
deficiency, could be divided into the following:-
o
equipment
o
processes
o
feedback
o
incentive and rewards
For individual factors the causes can be as classified as the
following
·
skills and knowledge
·
capacity of the individual
·
and motivation and expectations
We performed the post workshop survey from which we collated the
data on what the candidates considered the weaknesses of the workshop. This was
Kirkpatrick level I evaluation of learning.(Frye & Hemmer, 2012)
Cause Analysis diagram for performance gap in nerve repair workshop for residents in Orthopaedic
Surgery
Appendix 2
Individual factors
Motivation
Extrinsic
In this
domain the factors that would motivate the candidate to perform well in the
workshop will be external factors like a reward in the form of a positive
feedback and then visualising an improvement in his performance as a result of
the feedback. This is powerful in reinforcing behaviour and has been
incorporated in the workshop design. By using a model that simulates the real
life situation where digital nerve injuries are common in clinical practice, it
produces real world problems that can be solved with the workshop. This
contextualises the workshop to real-life clinical problems and therefore
motivates the candidate.(Bate, Hommes, Duvivier, & Taylor, 2013)
Intrinsic
Intrinsic factors that motivate the candidate to perform in this
workshop will be in accordance to Abraham Maslow’s hierarchy of needs. The
internal desire to perform well so that the candidate can produce a good nerve
repair on his patients and therefore produce good outcomes helps in the
self-actualisation and produces cognitive resonance for self-esteem and
recognition.(Taormina & Gao, 2013)
Capacity
Candidates must have the basic requirements of the ability to
perform basic surgical procedures like suturing and this has been taken account
as candidates have undergone basic surgical training. However some candidates
may not have had an opportunity to practice on microsurgical techniques and
therefore may not have the capacity to proceed in the workshop. Some candidates
may have physical impairments like tremors that may preclude them from
performing in this workshop.
Knowledge and skills
Most of the candidates would have had adequate knowledge as a result
of the lectures that have been provided for prior to the workshop. However the
skills of microsurgical techniques have not been introduced as most of them
have not done a basic microsurgery course. The first module in the workshop
does provide an opportunity for the acquisition of basic microsurgical
techniques and skills. However the time constraint does not allow for
deliberate practice sufficiently for all candidates to achieve a level of
competency that allows to progress to the more advance modules of the workshop.
This can be a factor that contributes to the low performance level in the final
assessment of the candidate.(Singh et al., 2013)
External factors
Equipment, resources and
support
One of the comments from the feedback survey indicated that some of
the instruments that were provided for did not meet up with the high quality
that is required for precision procedures. Initial workshops provided lectures
as part of the programme which were delivered synchronously by faculty members.
This did not allow for them to review the material asynchronously, on demand
and in a self-paced manner. As the workshop was run over two days is required
time to be taken from work to attend the workshop and some employers did not
provide support in the form of unrecorded leave.(Dubrowski, Brydges, Satterthwaite, Xeroulis, & Classen, 2012)
Data, information and
feedback
Performance of the individuals at the workshop was monitored by the
high faculty candidate ratio and provided adequate feedback in a timely manner
that allowed for improvements in the performance while at the workshop. Digital
recording of the summative assessment was provided to the candidates for review
in their own time for reflection and improvement. However there was no
opportunity for them to return to continue deliberate practice to improve the
performance due to the constraints of resources, as the workshop was only
conducted once a year.(Singh et al., 2013)
Consequences incentives
and rewards
Though there were certificates provided for the end of the cause to
provide recognition that they had attended and participated in the workshop
however there were no monetary incentives nor were there any accredited credits
towards postgraduate education. Continuing medical education points was
obtained and they still provide some incentive as part of the requirements for
the appraisal process of the clinicians on an annual basis. The facility to be
able to see Kirkpatrick level IV outcomes of learning by facilitating
submission of outcomes from real-life cases of nerve repair performed by the
individuals would be a great benefit and incentive to continue to improve
performance.
Discussion of various interventions:
Based on our analysis of the causes for the performance gap, the two
interventions that can be selected for improvement would be as follows.
For environmental factors, the provision of providing the knowledge
base for the workshop through online videos and lectures on a learning
management system (LMS) would allow for the support that the candidates need to
acquire the knowledge asynchronously, on demand and allowing self-paced
learning. This can be achieved by uploading all cognitive knowledge to a Moodle
website allowing for pre-workshop knowledge acquisition.
This will also resolve issues of employers not providing support
through conference leave to attend the workshop. This was simple to implement
as LMS was already available for utilisation to deliver the knowledge component
online. Videos of the workshop practical session can also be made available
online allowing the candidates to preview the curriculum and the skill
acquisition sequencing prior to attending the workshop. This will have a great
effect as it will reduce the time required to attending physically workshop, as
all knowledge component can be performed prior to the workshop. This will allow
dedicated time for deliberate practice to acquire skills during the workshop.
This will also be easy to implement as the LMSs is already available and is
merely converting the content into video and editing the instructional videos
to provide micro learning opportunity on the LMS.
The next intervention that can be used which will be a little bit
more difficult to implement, would be the creation of an integrated multimedia
kiosks with the resources for an on demand skill acquisition on-site. This is
to produce a workstation that is accessible by secure passes in the hospital
environment that would allow the candidates to use the instruments and
equipment and practice the various workshops on their own, under the guidance
of the instructional materials that would have been uploaded on the LMS. This would require some investment in funds
to design, develop and deploy these kiosks. This would automate this skill
acquisition workshop and reduce the cost as faculty need not be present in
real-time to supervise and assists the skill acquisition process.
Design of intervention
The provision of knowledge base through online learning objects.
In designing this strategy for improvement
in performance, would require the creation of multimedia rich and engaging
learning objects that would facilitate the acquisition of knowledge of a
practical procedure. Therefore this would require the production of videos with
narrations of the complex procedure of nerve repair. To facilitate the deep
learning and understanding of the concepts, there will be a need for
deconstructing the complex procedure to manageable chunks of information and
skill demonstration. These can be created using various e-learning authoring
tools like articulate storyline, to produce micro learning lectures consisting
of 10 minute videos of instruction. The appropriate sequencing of this micro
lecture will facilitate the understanding and retention of the knowledge and
skills required. This will allow for on demand and self-paced learning until
the candidates feels, comfortable that he has acquired the skills.
Creation of an integrated
multimedia kiosks
This will require support from
management in funding the physical creation of a workstation that would
consists of computer with access to Internet to allow for access of the
learning management system to review the online learning objects. This
workstation will also require the availability of table operating microscope
and microinstruments which are already available on-site. As the biological
model is perishable and requires logistic handling that may be resource
intensive, it will be prudent to just provide dry materials in the form of the first
module in the workshop where practice is performed on glove material.
This workstation will allow for
on demand, self-paced deliberate practice by the candidates in the own time and
increase the performance level until they have reached satisfactory level to
request for an assessment. This can be provided for by incorporating a web cam
in the workstation to record the summative assessment of the performance at the
candidates chosen time and then submitting that video to the faculty for
feedback and assessment.
Steps of implementing the intervention
Strategy 1
All available lectures that are in PowerPoint will need to be
converted to video with appropriate narrations using the e-learning authoring
tool like articulate storyline. These instructional videos will have to be structured
such that they are not more than 10 minutes in time and sequence to allow for
linear learning. These can be augmented with games and assessment of the
learning through quizzes and also engaging interests with the use of real life
case based discussion. These are then uploaded as resource materials onto the
Moodle learning management system. To encourage engagement and to assess the
learning, Sharable Content Object Reference Model (SCORM) compliant
learning objects can also be uploaded. This will allow for nonlinear learning
as those candidates with prior knowledge can choose the lectures and
instructions that are of interest to them. These can be made freely available
to all candidates and need to be communicated to them. The easy accessibility of
Moodle allows for the monitoring of the implementation of these learning
objects. It will also allow for the monitoring of the use and activities by
candidates of these learning objects. By incorporating activities like surveys
and feedback forms on the Moodle website, feedback from participants can be
obtained on the quality of these learning objects and thereby any review can be
performed by the faculty.
The possible problems of implementation of the strategy would be the
resources needed for the conversion of the current PowerPoint slides into
engaging videos. This will require time and effort and may not be sustainable
long-term for all the workshops that have been planned. One of the strategies
to overcome this would be to run workshops on production of digital assets for
the faculty so that subject matter experts like the faculty have the skill set
to be able to produce engaging digital objects to populate the learning
management system.(Khan, Khan, Dasgupta, & Ahmed, 2013)
Strategy 2
The creation of a multimedia workstation for on demand self paced
skill acquisition will require resources in the form of funding and personnel
to design a physical workstation that would incorporate a computer with
Internet access to the learning management system. It would require an
integrated web cam to facilitate recording of the skills being performed for
summative assessment. It also requires secure compartment for the storage of
microinstruments and sutures and the models for practice. This need to be in
close proximity and incorporated into the laboratory operating microscope. This
workstation will be required to be in a secure facility. Access has to be via
employee identification pass cards.
The barriers to implementation of the strategy would be the lack of
funding and space to incorporate the workstation to the laboratory microscope.
Also the availability of faculty to assess summative assessment may be overcome
by ensuring some form of incentive and reward is provided for faculty members
who are engaged in continuous assessment of trainees in skill acquisition. This
therefore would require buy in by senior management and restructuring of
performance management systems for faculty members.
Evaluation
As part of the summative evaluation of the interventions that have
been implemented to improve performance, the candidates can be encouraged to
maintain a portfolio of the cases that they have seen and how they have put
into practice what they acquired during the workshop. To further enhance the
confirmatory evaluation of the effect of the workshop, candidates can be
encouraged to forward evidence of having acquired and applied the skills from
the workshop in the workplace. This can be performed by the use of portfolios
on the learning management system for them to upload digital evidence of nerve
repair that they have performed in the workplace post workshop. This therefore
can be a Kirkpatrick level IV evidence of learning outcomes from a training
program.
References
Bate, E., Hommes, J., Duvivier, R., & Taylor, D. C. M. (2013).
Problem-based learning (PBL): Getting the most out of your students - Their
roles and responsibilities: AMEE Guide No. 84. Medical teacher. doi:10.3109/0142159X.2014.848269
Dubrowski, A., Brydges, R.,
Satterthwaite, L., Xeroulis, G., & Classen, R. (2012). Do not teach me
while I am working! American journal of surgery, 203(2), 253–257.
doi:10.1016/j.amjsurg.2010.08.020
Frye, A. W., & Hemmer, P. A.
(2012). Program evaluation models and related theories: AMEE guide no. 67. Medical
teacher, 34(5), e288–299. doi:10.3109/0142159X.2012.668637
Khan, N., Khan, M. S., Dasgupta, P.,
& Ahmed, K. (2013). The surgeon as educator: fundamentals of faculty training
in surgical specialties. BJU international, 111(1), 171–178.
doi:10.1111/j.1464-410X.2012.11336.x
Mandal, K., & Banerjee, C. K.
(2012). An Empirical Identification of Performance Gap in Engineering Education
Program from the Perspective of Stakeholders. International Journal of
Trade, Economics & Finance, 3(4), 281.
Singh, P., Aggarwal, R., Pucher, P.
H., Duisberg, A. L., Arora, S., & Darzi, A. (2013). Defining quality in
surgical training: perceptions of the profession. American journal of
surgery. doi:10.1016/j.amjsurg.2013.07.044
Taormina, R. J., & Gao, J. H.
(2013). Maslow and the motivation hierarchy: measuring satisfaction of the
needs. The American journal of psychology, 126(2), 155–177.
Bate, E., Hommes, J., Duvivier, R., & Taylor, D. C. M. (2013).
Problem-based learning (PBL): Getting the most out of your students - Their
roles and responsibilities: AMEE Guide No. 84. Medical teacher.
doi:10.3109/0142159X.2014.848269
Dubrowski, A., Brydges, R., Satterthwaite,
L., Xeroulis, G., & Classen, R. (2012). Do not teach me while I am working!
American journal of surgery, 203(2), 253–257.
doi:10.1016/j.amjsurg.2010.08.020
Frye, A. W., & Hemmer, P. A.
(2012). Program evaluation models and related theories: AMEE guide no. 67. Medical
teacher, 34(5), e288–299. doi:10.3109/0142159X.2012.668637
Khan, N., Khan, M. S., Dasgupta, P.,
& Ahmed, K. (2013). The surgeon as educator: fundamentals of faculty
training in surgical specialties. BJU international, 111(1), 171–178.
doi:10.1111/j.1464-410X.2012.11336.x
Mandal, K., & Banerjee, C. K.
(2012). An Empirical Identification of Performance Gap in Engineering Education
Program from the Perspective of Stakeholders. International Journal of
Trade, Economics & Finance, 3(4), 281.
Singh, P., Aggarwal, R., Pucher, P.
H., Duisberg, A. L., Arora, S., & Darzi, A. (2013). Defining quality in
surgical training: perceptions of the profession. American journal of
surgery. doi:10.1016/j.amjsurg.2013.07.044
Taormina, R. J., & Gao, J. H.
(2013). Maslow and the motivation hierarchy: measuring satisfaction of the
needs. The American journal of psychology, 126(2), 155–177.
Appendix
1
Skill workshop outcomes and Rubrics for
assessment
Practical 2 Micro suturing practices
You should be able
· Place a 9/0 needle at various angles through a cut in the rubber glove
· Place a 9/0 needle using a back hand technique
· Perform a double throw on a 9/0 suture followed by a single throw and
ensure the knots are square and locked in full engaged position.
Here you can see that the learning outcome has been very clearly
specified for this workshop.
The activity that will be used for this specific outcome would be
the use of video to explain the procedures of placing sutures in the glove.
Course material will be provided for students to prepare prior to the workshop.
Once the skill has been observed on the video, the students then watch a
real-life performance of the skill by an expert. Then the students perform the
skill on the model rubber glove. They will repeat the process until they feel
they have reached a level of competency for assessment to be performed.
So in creating a blueprint for this program, on one side we have the
learning outcome which will be placing a 9/0 needle at various angles through a
cut in the rubber glove. The activities that will go with this learning outcome
will include the reading of course material, to watching of video and watching
of an expert perform the procedure. And the assessment tools for this would be
direct observation of procedure by an expert. We propose that this assessment
can be recorded and posted online for assessment asynchronously by the expert.
The rubrics for this assessment will be placed on the grading of the
quality of the repair that has been performed. Detail descriptors are used to
clearly identify the level of performance without ambiguity for both the
student and the assessor.
Assessment for micro suturing
Placement of suture
1 poorly spaced and angled ,and not square in more than 80%
2 space and angle of sutures satisfactory in less than 40%
3 Space and angle of sutures satisfactory in 50 to 60% of sutures
4 Space and angle of sutures between 60 to 80% of sutures
5 Well-spaced and angled and all sutures square and locked in more than
80% of sutures.
Tension
1 Gaps in incision line or overlapping of edges in more than 80%
2 Gaps in incision line or overlapping of edges in less than 60%
3 Gaps in incision line or overlapping of edges satisfactory between 50%
to 40%
4 Gaps in incision line or overlapping of edges in less than 20%
5 No gaps, well tensioned with good coaption of edges without crimping
Knots
1 Inappropriate length and poorly located in more than 80%
2 Appropriate length and poorly located in less than 40%
3 Appropriate length and poorly located in 50 to 60% of sutures
4 Appropriate length and poorly located in 60 to 80% of sutures
5 . Well cut ends with placement of
knot in secure place with no potential entanglement.
The assessment that will be performed will be based on
the candidate suturing a fixed number of sutures across a gap in the rubber
glove. The quality of the suturing is then measured using these rubrics and
this ensures that the assessment is valid both informs of content and
context and reliable between different assesses because the descriptors are
clear.
No comments:
Post a Comment