Friday 11 March 2016

Human Performance Technology (HPT) in surgical education - my experience



In a training program to teach orthopaedic surgery residents on the microsurgery skills of repairing the nerves of the hand, the current success rate is not as expected and the resources that are used to run this workshop is significant.

The problem is,

 How to increase the rate of successful acquisition of the skill of nerve repair in this high fidelity skill workshop.

Benefits that can be gained by solving the problem           

If the success rate can be increased, the number of trained personnel will be higher the number of successful nerve repair done in the hospital will be higher and therefore reducing the number re operations and secondary reconstructive procedures with better outcomes for the patients.(Mandal & Banerjee, 2012)

What is expected when the problem is solved or performance is improved?  

If the success rate of skill acquisition in the workshop can be increased the following can be expected:-

1.     number of surgeon competent to perform nerve surgery will be higher
2.     the number of successful nerve repair done in the hospital will be higher 
3.     reduction in the number re operations for nerve surgery
4.     secondary reconstructive procedures for nerve injuries will be less
5.     Better outcomes for patients with nerve injury.

Determination of the performance gap:

The learning outcome for this highly targeted workshop was the ability by the candidates to perform microsurgical repair of a nerve in a biological model. The assessment of the outcome had been developed with the use of an objective structured rubrics that had been created. (Appendix 1)

This workshop runs for two days with 10 candidates and approximately 4 faculty members. Estimated cost for the workshop is approximately USD $50,000 per day. Thus poor outcomes for the workshop makes it cost ineffective to continue to run the workshop.

Assessments were provided as a formative type during the workshop with immediate feedback so as the candidates were able to learn effectively with this formative assessment. This was provided for by the faculty and thus the higher faculty candidate ratio. Following repeated and deliberate practice of the procedure on the model, they were then assessed summatively which was recorded digitally and the time taken to perform the task was also recorded. The final assessment of this exercise was performed by three independent assessors who were senior surgeons using the rubrics in appendix 1. The rubrics were developed according to standard protocol used in objective structured clinical evaluation. As can be seen in the rubrics the descriptors for the various levels of performance were clearly stated.

From the two workshops that we had conducted the number of candidates who had a score of four upon five on the Likert score for the quality of repair was 20%. This was the actual performance of candidates in this resource intensive workshop that we were conducting. We had produce measurement tools that were structured and had internal validity to conduct an assessment of the quality of repair. In the ideal world we would have expected 100% of the candidates being able to perform a good repair according to our rubrics. Thus currently this workshop was producing a performance gap of 80%.

Measurements of improvement in performance of the candidates will be conducted using the same rubrics and therefore providing reliability between the two measurements.

Factors affecting performance:

Areas that will affect performance in this workshop can be divided into
·       environmental factors and
·       individual factors
For environmental factors that can be a cause for performance deficiency, could be divided into the following:-
o   equipment
o   processes
o   feedback
o   incentive and rewards
For individual factors the causes can be as classified as the following
·       skills and knowledge
·       capacity of the individual
·       and motivation and expectations
We performed the post workshop survey from which we collated the data on what the candidates considered the weaknesses of the workshop. This was Kirkpatrick level I evaluation of learning.(Frye & Hemmer, 2012)

Cause Analysis diagram for performance gap in nerve repair workshop for residents in Orthopaedic Surgery
Appendix 2

Individual factors
            Motivation
                        Extrinsic
                        In this domain the factors that would motivate the candidate to perform well in the workshop will be external factors like a reward in the form of a positive feedback and then visualising an improvement in his performance as a result of the feedback. This is powerful in reinforcing behaviour and has been incorporated in the workshop design. By using a model that simulates the real life situation where digital nerve injuries are common in clinical practice, it produces real world problems that can be solved with the workshop. This contextualises the workshop to real-life clinical problems and therefore motivates the candidate.(Bate, Hommes, Duvivier, & Taylor, 2013)
                        Intrinsic
Intrinsic factors that motivate the candidate to perform in this workshop will be in accordance to Abraham Maslow’s hierarchy of needs. The internal desire to perform well so that the candidate can produce a good nerve repair on his patients and therefore produce good outcomes helps in the self-actualisation and produces cognitive resonance for self-esteem and recognition.(Taormina & Gao, 2013)
            Capacity
Candidates must have the basic requirements of the ability to perform basic surgical procedures like suturing and this has been taken account as candidates have undergone basic surgical training. However some candidates may not have had an opportunity to practice on microsurgical techniques and therefore may not have the capacity to proceed in the workshop. Some candidates may have physical impairments like tremors that may preclude them from performing in this workshop.

            Knowledge and skills
Most of the candidates would have had adequate knowledge as a result of the lectures that have been provided for prior to the workshop. However the skills of microsurgical techniques have not been introduced as most of them have not done a basic microsurgery course. The first module in the workshop does provide an opportunity for the acquisition of basic microsurgical techniques and skills. However the time constraint does not allow for deliberate practice sufficiently for all candidates to achieve a level of competency that allows to progress to the more advance modules of the workshop. This can be a factor that contributes to the low performance level in the final assessment of the candidate.(Singh et al., 2013)

External factors
Equipment, resources and support
One of the comments from the feedback survey indicated that some of the instruments that were provided for did not meet up with the high quality that is required for precision procedures. Initial workshops provided lectures as part of the programme which were delivered synchronously by faculty members. This did not allow for them to review the material asynchronously, on demand and in a self-paced manner. As the workshop was run over two days is required time to be taken from work to attend the workshop and some employers did not provide support in the form of unrecorded leave.(Dubrowski, Brydges, Satterthwaite, Xeroulis, & Classen, 2012)

Data, information and feedback
Performance of the individuals at the workshop was monitored by the high faculty candidate ratio and provided adequate feedback in a timely manner that allowed for improvements in the performance while at the workshop. Digital recording of the summative assessment was provided to the candidates for review in their own time for reflection and improvement. However there was no opportunity for them to return to continue deliberate practice to improve the performance due to the constraints of resources, as the workshop was only conducted once a year.(Singh et al., 2013)

Consequences incentives and rewards
Though there were certificates provided for the end of the cause to provide recognition that they had attended and participated in the workshop however there were no monetary incentives nor were there any accredited credits towards postgraduate education. Continuing medical education points was obtained and they still provide some incentive as part of the requirements for the appraisal process of the clinicians on an annual basis. The facility to be able to see Kirkpatrick level IV outcomes of learning by facilitating submission of outcomes from real-life cases of nerve repair performed by the individuals would be a great benefit and incentive to continue to improve performance.

Discussion of various interventions:
Based on our analysis of the causes for the performance gap, the two interventions that can be selected for improvement would be as follows.

For environmental factors, the provision of providing the knowledge base for the workshop through online videos and lectures on a learning management system (LMS) would allow for the support that the candidates need to acquire the knowledge asynchronously, on demand and allowing self-paced learning. This can be achieved by uploading all cognitive knowledge to a Moodle website allowing for pre-workshop knowledge acquisition.
This will also resolve issues of employers not providing support through conference leave to attend the workshop. This was simple to implement as LMS was already available for utilisation to deliver the knowledge component online. Videos of the workshop practical session can also be made available online allowing the candidates to preview the curriculum and the skill acquisition sequencing prior to attending the workshop. This will have a great effect as it will reduce the time required to attending physically workshop, as all knowledge component can be performed prior to the workshop. This will allow dedicated time for deliberate practice to acquire skills during the workshop. This will also be easy to implement as the LMSs is already available and is merely converting the content into video and editing the instructional videos to provide micro learning opportunity on the LMS.

The next intervention that can be used which will be a little bit more difficult to implement, would be the creation of an integrated multimedia kiosks with the resources for an on demand skill acquisition on-site. This is to produce a workstation that is accessible by secure passes in the hospital environment that would allow the candidates to use the instruments and equipment and practice the various workshops on their own, under the guidance of the instructional materials that would have been uploaded on the LMS.  This would require some investment in funds to design, develop and deploy these kiosks. This would automate this skill acquisition workshop and reduce the cost as faculty need not be present in real-time to supervise and assists the skill acquisition process.


Design of intervention

The provision of knowledge base through online learning objects.

In designing this strategy for improvement in performance, would require the creation of multimedia rich and engaging learning objects that would facilitate the acquisition of knowledge of a practical procedure. Therefore this would require the production of videos with narrations of the complex procedure of nerve repair. To facilitate the deep learning and understanding of the concepts, there will be a need for deconstructing the complex procedure to manageable chunks of information and skill demonstration. These can be created using various e-learning authoring tools like articulate storyline, to produce micro learning lectures consisting of 10 minute videos of instruction. The appropriate sequencing of this micro lecture will facilitate the understanding and retention of the knowledge and skills required. This will allow for on demand and self-paced learning until the candidates feels, comfortable that he has acquired the skills.

Creation of an integrated multimedia kiosks

This will require support from management in funding the physical creation of a workstation that would consists of computer with access to Internet to allow for access of the learning management system to review the online learning objects. This workstation will also require the availability of table operating microscope and microinstruments which are already available on-site. As the biological model is perishable and requires logistic handling that may be resource intensive, it will be prudent to just provide dry materials in the form of the first module in the workshop where practice is performed on glove material.

This workstation will allow for on demand, self-paced deliberate practice by the candidates in the own time and increase the performance level until they have reached satisfactory level to request for an assessment. This can be provided for by incorporating a web cam in the workstation to record the summative assessment of the performance at the candidates chosen time and then submitting that video to the faculty for feedback and assessment.

Steps of implementing the intervention

Strategy 1
All available lectures that are in PowerPoint will need to be converted to video with appropriate narrations using the e-learning authoring tool like articulate storyline. These instructional videos will have to be structured such that they are not more than 10 minutes in time and sequence to allow for linear learning. These can be augmented with games and assessment of the learning through quizzes and also engaging interests with the use of real life case based discussion. These are then uploaded as resource materials onto the Moodle learning management system. To encourage engagement and to assess the learning, Sharable Content Object Reference Model (SCORM) compliant learning objects can also be uploaded. This will allow for nonlinear learning as those candidates with prior knowledge can choose the lectures and instructions that are of interest to them. These can be made freely available to all candidates and need to be communicated to them. The easy accessibility of Moodle allows for the monitoring of the implementation of these learning objects. It will also allow for the monitoring of the use and activities by candidates of these learning objects. By incorporating activities like surveys and feedback forms on the Moodle website, feedback from participants can be obtained on the quality of these learning objects and thereby any review can be performed by the faculty.

The possible problems of implementation of the strategy would be the resources needed for the conversion of the current PowerPoint slides into engaging videos. This will require time and effort and may not be sustainable long-term for all the workshops that have been planned. One of the strategies to overcome this would be to run workshops on production of digital assets for the faculty so that subject matter experts like the faculty have the skill set to be able to produce engaging digital objects to populate the learning management system.(Khan, Khan, Dasgupta, & Ahmed, 2013)

Strategy 2
The creation of a multimedia workstation for on demand self paced skill acquisition will require resources in the form of funding and personnel to design a physical workstation that would incorporate a computer with Internet access to the learning management system. It would require an integrated web cam to facilitate recording of the skills being performed for summative assessment. It also requires secure compartment for the storage of microinstruments and sutures and the models for practice. This need to be in close proximity and incorporated into the laboratory operating microscope. This workstation will be required to be in a secure facility. Access has to be via employee identification pass cards.

The barriers to implementation of the strategy would be the lack of funding and space to incorporate the workstation to the laboratory microscope. Also the availability of faculty to assess summative assessment may be overcome by ensuring some form of incentive and reward is provided for faculty members who are engaged in continuous assessment of trainees in skill acquisition. This therefore would require buy in by senior management and restructuring of performance management systems for faculty members.

Evaluation
As part of the summative evaluation of the interventions that have been implemented to improve performance, the candidates can be encouraged to maintain a portfolio of the cases that they have seen and how they have put into practice what they acquired during the workshop. To further enhance the confirmatory evaluation of the effect of the workshop, candidates can be encouraged to forward evidence of having acquired and applied the skills from the workshop in the workplace. This can be performed by the use of portfolios on the learning management system for them to upload digital evidence of nerve repair that they have performed in the workplace post workshop. This therefore can be a Kirkpatrick level IV evidence of learning outcomes from a training program.





References

Bate, E., Hommes, J., Duvivier, R., & Taylor, D. C. M. (2013). Problem-based learning (PBL): Getting the most out of your students - Their roles and responsibilities: AMEE Guide No. 84. Medical teacher. doi:10.3109/0142159X.2014.848269
Dubrowski, A., Brydges, R., Satterthwaite, L., Xeroulis, G., & Classen, R. (2012). Do not teach me while I am working! American journal of surgery, 203(2), 253–257. doi:10.1016/j.amjsurg.2010.08.020
Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical teacher, 34(5), e288–299. doi:10.3109/0142159X.2012.668637
Khan, N., Khan, M. S., Dasgupta, P., & Ahmed, K. (2013). The surgeon as educator: fundamentals of faculty training in surgical specialties. BJU international, 111(1), 171–178. doi:10.1111/j.1464-410X.2012.11336.x
Mandal, K., & Banerjee, C. K. (2012). An Empirical Identification of Performance Gap in Engineering Education Program from the Perspective of Stakeholders. International Journal of Trade, Economics & Finance, 3(4), 281.
Singh, P., Aggarwal, R., Pucher, P. H., Duisberg, A. L., Arora, S., & Darzi, A. (2013). Defining quality in surgical training: perceptions of the profession. American journal of surgery. doi:10.1016/j.amjsurg.2013.07.044
Taormina, R. J., & Gao, J. H. (2013). Maslow and the motivation hierarchy: measuring satisfaction of the needs. The American journal of psychology, 126(2), 155–177.
Bate, E., Hommes, J., Duvivier, R., & Taylor, D. C. M. (2013). Problem-based learning (PBL): Getting the most out of your students - Their roles and responsibilities: AMEE Guide No. 84. Medical teacher. doi:10.3109/0142159X.2014.848269
Dubrowski, A., Brydges, R., Satterthwaite, L., Xeroulis, G., & Classen, R. (2012). Do not teach me while I am working! American journal of surgery, 203(2), 253–257. doi:10.1016/j.amjsurg.2010.08.020
Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical teacher, 34(5), e288–299. doi:10.3109/0142159X.2012.668637
Khan, N., Khan, M. S., Dasgupta, P., & Ahmed, K. (2013). The surgeon as educator: fundamentals of faculty training in surgical specialties. BJU international, 111(1), 171–178. doi:10.1111/j.1464-410X.2012.11336.x
Mandal, K., & Banerjee, C. K. (2012). An Empirical Identification of Performance Gap in Engineering Education Program from the Perspective of Stakeholders. International Journal of Trade, Economics & Finance, 3(4), 281.
Singh, P., Aggarwal, R., Pucher, P. H., Duisberg, A. L., Arora, S., & Darzi, A. (2013). Defining quality in surgical training: perceptions of the profession. American journal of surgery. doi:10.1016/j.amjsurg.2013.07.044
Taormina, R. J., & Gao, J. H. (2013). Maslow and the motivation hierarchy: measuring satisfaction of the needs. The American journal of psychology, 126(2), 155–177.
  
Appendix

1
Skill workshop outcomes and Rubrics for assessment

Practical 2 Micro suturing practices
You should be able
·         Place a 9/0 needle at various angles through a cut in the rubber glove
·         Place a 9/0 needle using a back hand technique
·         Perform a double throw on a 9/0 suture followed by a single throw and ensure the knots are square and locked in full engaged position.
Here you can see that the learning outcome has been very clearly specified for this workshop.
 The activity that will be used for this specific outcome would be the use of video to explain the procedures of placing sutures in the glove. Course material will be provided for students to prepare prior to the workshop. Once the skill has been observed on the video, the students then watch a real-life performance of the skill by an expert. Then the students perform the skill on the model rubber glove. They will repeat the process until they feel they have reached a level of competency for assessment to be performed.
So in creating a blueprint for this program, on one side we have the learning outcome which will be placing a 9/0 needle at various angles through a cut in the rubber glove. The activities that will go with this learning outcome will include the reading of course material, to watching of video and watching of an expert perform the procedure. And the assessment tools for this would be direct observation of procedure by an expert. We propose that this assessment can be recorded and posted online for assessment asynchronously by the expert.
The rubrics for this assessment will be placed on the grading of the quality of the repair that has been performed. Detail descriptors are used to clearly identify the level of performance without ambiguity for both the student and the assessor.
Assessment for micro suturing
Placement of suture
1              poorly spaced and angled ,and not square in more than 80%
2              space and angle of sutures  satisfactory in less than 40%
3              Space and angle of sutures satisfactory in 50 to 60% of sutures
4              Space and angle of sutures between 60 to 80% of sutures
5              Well-spaced and angled and all sutures square and locked in more than 80% of sutures.
Tension
1        Gaps in incision line or overlapping of edges in more than 80%
2        Gaps in incision line or overlapping of edges in less than 60%
3        Gaps in incision line or overlapping of edges satisfactory between 50% to 40%
4        Gaps in incision line or overlapping of edges in less than 20%
5        No gaps, well tensioned with good coaption of edges without crimping
Knots
1        Inappropriate length and poorly located in more than 80%
2         Appropriate length and poorly located in less than 40%
3        Appropriate length and poorly located in 50 to 60% of sutures
4        Appropriate length and poorly located in 60 to 80% of sutures
5        . Well cut ends with placement of knot in secure place with no potential entanglement.
 Here rubrics have been created for each domain with the marking scheme that are clear.
The assessment that will be performed will be based on the candidate suturing a fixed number of sutures across a gap in the rubber glove. The quality of the suturing is then measured using these rubrics and this ensures that the assessment is valid both informs of content and context and reliable between different assesses because the descriptors are clear.





No comments:

Post a Comment