Abstract
Introduction
Professionalism has been identified as a core domain in medical education, with the Medical Council of New Zealand (MCNZ), 1 for example, stating that “Patients are entitled to good doctors. Good doctors make the care of their patients their first concern: they are competent, keep their knowledge and skills up to date, establish and maintain good relationships with patients and colleagues, are honest and trustworthy, and act ethically” (p. 6). The importance of professionalism has been emphasized in the New Zealand Curriculum Framework for Prevocational Medical Training recently adopted by the MCNZ. This document outlines the learning outcomes that are to be substantively completed by the end of the second postgraduate year in practice.
Professionalism has been used as a central concept in creating a new domain of learning in the medical program at the University of Auckland called the Personal and Professional Skills (PPS) domain. This domain runs longitudinally over the five years of the program (Years 2–6; Year 1 comprises a common Health Science selection year). The principal means of assessment for this domain is through the compilation of a portfolio in each year that reflects the learning outcomes for the five themes of the domain. The domain and the portfolio were introduced to Years 2 and 4 of the program in 2013, Years 3 and 5 in 2014, and Year 6 in 2015.
The five themes of learning included in the PPS domain are Professionalism and Reflective Practice, Ethics and the Law, Cultural Competence, Health and Well-being, and Learning and Teaching. These themes were consolidated in 2014 through a systematic review of medical literature on topics included in the PPS domain of learning. The analysis confirmed the importance of inclusion of these themes and also highlighted any significant gaps in medical curricula internationally.
Assessment in this domain of learning can be challenging, as it requires a judgment being made on affective and qualitative elements, such as attitudes, professionalism, the ability for critical reflection, personal growth and development, and how these translate into clinical practice. These are not easily assessed by traditional assessment methods used to measure more objective and standardized medical knowledge.2–6 However, a systematic review of medical education literature on the use of portfolios found that portfolios can support both learning and assessment of these elements in undergraduate medical programs. 7
The assessment principles adopted for the PPS domain include the need for more than one overall method of assessment, along with multiple, small samples of evidence over time to provide reliability and allow the assessment of progress. While the assessment for the domain consists of three primary elements to meet the first of these requirements (direct observation by clinical staff, assignments, and portfolio), the assessment event that carries the majority weighting is the portfolio (refer to the grading rubric for Year 4 shown in Table 1). If students fail the portfolio, they are given one opportunity for remediation/resubmission before failing the year. The portfolio is student centered and driven, enables student learning through assessment, 2 and allows the compilation and presentation of multiple pieces of evidence to demonstrate how the domain learning outcomes have been achieved. Martin-Kneip (cited by Friedman Ben David et al 2 ) highlights that the “… collection represents a personal investment on the part of the student–-an investment that is evident through the student's participation in the selection of the contents, the criteria for selection, the criteria for judging the merit of the collection and the student's self-reflection” (p. 536). While the use of portfolios in a postgraduate medical context is reasonably well established,2,3,7–10 use in undergraduate medical programs is a more recent phenomenon, with their value becoming increasingly recognized.2,4,7,11–14 In particular, they are valued for their ability to promote reflection.2–4,7,11–15 There are useful discussions and guides available in the literature that identify both strengths and limitations in their use.2–4,7,12,14–17 This includes reference to the difficulty of assessing the work presented, including self-assessment and the ability to reflect critically.3,16 The portfolio assessment process needs to accept the subjective, process-based, and progressive nature of the work being presented, which has driven the development of an assessment matrix at the University of Auckland that focuses on process rather than specific content. This is a qualitative approach that requires the assessor to make a professional judgment. 12 Friedman Ben David et al note that the ability to reflect is increasingly being recognized as an important component of medical professionalism 2 on the grounds that it requires professionals to consider their own actions from the viewpoint of critically assessing what worked, what did not work, and how that action could be improved for the best patient outcome. Kalet et al 5 link reflection to experiential learning, commenting that it “translates the experience of clinical practice into learning and is a crucial intellectual task in professional competency” (p. 1066). Others3,13 link reflection to lifelong learning, maintaining that it provides the foundation for best practice with associated links to patient benefit. According to Driessen et al, 7 both medical students and doctors are limited in their ability to engage in reflection and self-assessment, which highlights the importance of including opportunities to develop this important ability in medical programs.
Grading rubric for Year 4 PPS domain.
The format for the portfolio at the University of Auckland has been developed around the five themes of the PPS domain, whereby the students demonstrate how they have met the learning outcomes for each theme. In addition, they are required to include a table of contents (developed from a portfolio plan), a conclusion, and are recommended to include a curriculum vitae, so that it becomes a growing document that can be used in the future. Templates are provided for optional use for recording evidence, for example, for significant learning events and reading logs.
The introduction of portfolios has presented a challenge to the perceptions and learning of some students in what has previously been a fairly traditional medical school curriculum. However, the standard of student work assessed to date in most cases has been outstanding and has shown evidence of personal and professional development that has not been previously evidenced through assessment. This article presents the process for introducing the portfolio, including the challenges, strengths and limitations, evaluation of data to date, and the ways in which student feedback has been addressed.
Method
The implementation process employed a cyclical model that was designed to evaluate and improve the use of the portfolio, that could form the basis of the first cycle of action research that will be continued in the future to improve the initiative (Fig. 1). During 2013, the introduction of the portfolio was monitored, and changes were made in 2014. In 2014, feedback from students indicated the need for further modifications, which are currently being implemented, along with the introduction of an electronic platform.
Portfolio evaluation model.
Student feedback was ascertained from the generic end-of-year course evaluation, which included questions relating to the PPS domain. The number of students in each year varied from 199–260 and there was an average response rate of 50%. In addition, the Auckland University Medical Students Association conducted a student evaluation for the purpose of writing a report for the ongoing medical program accreditation, from which key issues were collated and forwarded to the PPS domain coordinators. The eight portfolio assessors also provided suggestions for improvement at the end of 2013 and 2014. Evaluation feedback, modifications, and qualitative data derived from the portfolios are presented in the Findings section.
Findings
Due to the qualitative nature of the feedback, it was analyzed inductively, through a cross-sectional thematic analysis that sought to ascertain the strongest themes from across the three sources of feedback. The key points presented are listed in Table 2. Qualitative comments directly included in student portfolios have been included in the Discussion section. Given that the PPS coordinators reflexively interpreted the data through their own lens of experience in the portfolio process, which brings with it both strengths and limitations, it is planned to conduct a full, independent evaluation after five years of implementation, at the end of 2017.
Summary of feedback.
Table 3 indicates the modifications that were made as a direct result of this feedback.
Modifications made.
Discussion
It was anticipated that students may have difficulty with personally reflective learning and assessment, as this is a substantial change to previous forms of assessment. However, many students related to the portfolio extremely well, noting that it is the only part of the curriculum in which they can be creative, develop their critical reflection, and show a different aspect of who they are (Table 4). Others appeared to be frustrated by the assessment, perhaps feeling outside of their comfort zone and/or not convinced of the relevance of PPS content or portfolio process, and this resulted in some angry or defensive responses. Anxiety created by a new form of learning or assessment is to be expected.2,17 Even though negative attitudes and misconceptions about reflective learning have been noted in the literature,
17
it was disappointing to experience the degree of resistance shown by some. As suggested by Ross et al,
17
this may reflect a broader culture in medicine that does not support reflective learning, or understand its importance. There appeared to be a general lack of understanding about the value of the PPS domain and reflective practice. Some students resented
excerpts from 2014 portfolios.
Much of the negative feedback (refer feedback from students, Table 2) came from the more senior students who had been exposed to the program before the implementation of the PPS domain and portfolio. Therefore, we hope this will change over time, particularly with improved orientation sessions highlighting the importance of the PPS domain and the skills required to develop a portfolio for their future careers as doctors. Furthermore, Kalet et al 5 discuss that it is reasonable to expect that while some students may not see the relevance of reflective writing early in their training, it may prove useful to them later in their careers. They conclude that while student satisfaction is important, it should not be the only measure used when implementing and evaluating this form of assessment.
Looking critically at the process we have implemented, it appears to meet most of the suggestions for best practice explored in the literature. The strengths of our approach include the following.
Assessment of learning outcomes
The portfolio summatively assesses evidence of attainment of the learning outcomes of the PPS domain longitudinally across the medical program, allowing the assessment of aspects of the program not previously assessed and the assessment of progression.2,6,7,14 Therefore, students must pass the portfolio each year in order to progress to the following year of their program.
Emphasis on adult principles of learning
The portfolio provides authentic assessment of the past and present personal, academic, and clinical experiences that contribute to the themes of the PPS domain.2–5,9,12,15,16 Authentic assessment encourages students to become responsible for their own learning and focus on the development of lifelong learning skills.2,6,9,15 In order to maintain authenticity, it is important to uphold a personal and individualized approach that elicits the real experience of the learner, helps them to consolidate their learning, and make connections between theory and practice.2,3,5,15 According to Kalet et al,
5
authentic assessment emphasizes “the process as well as the products of learning“ (p. 1066), which is reflected in our marking rubric. Interestingly, some students doubt the authenticity of entries made by some of their peers, maintaining that they could
Authentic assessment is one of the key principles of adult learning. The use of a portfolio to build on the learning value of student experience through critical reflection emphasizes an adult model of learning. According to Mathers et al, 10 this model is more likely to result in deep learning than courses driven by direct teaching input with test/assignment-focused assessment. Challis 3 outlines the characteristics that lead to deep learning, from which we can draw additional portfolio characteristics of active rather than passive learning3,10 and the integrated nature of the way knowledge is presented, rather than the more traditional way of seeing knowledge as independent pieces of content. 3 Portfolio-based learning is also said to increase self-knowledge and confidence of students. 10
Mezirow (cited by Challis 3 ) defines androgeny as “an organised and sustained effort to assist adults to learn in a way that enhances their capability to function as self-directed learners” (p. 372). Central to this is the ability to value, reflect, and learn from experience, 16 with experiential and self-directed approaches being said to foster patterns of lifelong learning.3,5,6,13 As outlined by Challis, 3 students need to be able to engage in actual experience, critically reflect, conceptualize or make meaning from this experience, 18 and then apply the concepts/meaning to new situations. Driessen et al 15 “regard reflection as a cyclic process of self-regulation” (p. 1230) and maintain that it is a condition for professional development. To enhance this process, we encourage students to include a concluding section where they can summarize what they have learned personally and professionally over the year and how that translates into their goals for the following year. This also leads into the Year 6 transition to the more goal-focused professional development plan, which is a requirement of the MCNZ for junior doctors.
Summative assessment
The portfolio is suitable for summative assessment, 14 based on professional qualities, reflection, and the ability to demonstrate that learning outcomes have been met, rather than purely on content. According to Driessen et al, it needs to be summative if its status is to be maintained in the eyes of students, who tend to be assessment driven, 7 and it means that they can be rewarded for their effort in producing a high-quality portfolio. The AMEE Medical Education Guide 2 suggests that if it assesses multiple competencies, general standards should be developed rather than being highly specific. Likewise, the systematic review by Driessen et al 7 revealed that the use of global criteria with rubrics has a positive impact on interrater agreement, both of which support the process nature of our marking rubric. To assess in an interpretive manner means accepting the subjectivity of the personal and often creative material included,3,4,16 and this is supported by the marking rubric, which has been adapted from the REFLECT rubric. 19
While the portfolio has been designed to be a criterion rather than norm referenced, 12 the rubric allows for the allocation of a distinction grade for excelling portfolios, which provides a point of discrimination for the overall calculation of distinction for the domain grade for the year. This signals to students the importance of the PPS domain, alongside applied medical knowledge and their clinical attachments. All portfolios with an indication of fail, borderline pass, high pass, or distinction are moderated by the PPS domain coordinators to ensure consistency of standards at grade cut points.
Structure
A portfolio structure that is too standardized is said to decrease authenticity and validity,
2
which was one of the drivers for the structure chosen for the PPS portfolio. While we require the students to submit a minimum of two pieces of evidence for each of the five themes of the domain, they are given multiple suggestions in the portfolio guidelines as to what types of evidence they may include. These are not tightly defined–-it is left to the students to select pieces of evidence and reflection that illustrate how they meet the learning outcomes for each of the themes of the domain. It is stressed that their work is assessed on the
Variety of evidence
The portfolio may include a wide variety of forms of evidence over time and allows the students to be creative and personalize their learning.2,6,7 For example, many students express their reflections in poetry or art, particularly when providing evidence for the Health and Well-being theme. It is not intended that the portfolio is purely a collection of evidence, rather each piece is accompanied by reflection on its relevance to the PPS learning outcomes and how it has led to their personal or professional development. 3 That is, they need to articulate what they have learned through the process of critical reflection. 8 This engages them actively in the process of self-assessment and also encourages them to synthesize their learning from activities they have been engaged in to demonstrate their personal or professional development over time. 9
Finally, this discussion turns to the arguments that arise with regard to validity, reliability, and standardization. Our stance is that a highly structured portfolio diminishes its strength in assessing personal and professional development. However, the trade-off for this is that it means the content of the portfolios cannot be standardized for the traditional
As stated previously, we accept the intrinsically personal and subjective nature of the material presented, grading it according to process, not content. A degree of standardization is achieved through the requirement for students to provide evidence of meeting the learning outcomes for each theme of the domain, as reflected in the structure of the portfolio. The personalized nature of the portfolio, however, contributes directly to its authenticity and, therefore, validity.2,7 Challis 3 reminds us that the subjectivity always involved in assessing portfolios is not a disadvantage, as our aim is to decide “whether appropriate learning has taken place and has been demonstrated, in accordance with the development needs of the learner” (p. 376).
There are two primary areas that still need to be addressed before a larger scale independent evaluation of the portfolio is conducted. First, it is clear in the literature and from student feedback that it would be valuable to have portfolio mentors who can be available to assist with the process of compiling their portfolios. In 2015, training was introduced for the small group of tutors who are involved in teaching the PPS domain in Years 2 and 3, to enable them to work with their groups of students in a more consistent and informed manner. However, in Years 4–6, we have so far had mixed success. Portfolio drop-in times have been scheduled where students can bring questions or problems to the PPS domain coordinators; however, there has not been a high level of uptake to date. A mentoring scheme is being explored for students wishing to process difficult clinical experiences and has been introduced primarily as a pastoral care initiative. However, portfolio feedback in 2014 indicated that for some students, the process of writing about a challenging clinical experience raised their awareness of their emotions and vulnerability, thus increasing their anxiety. Some students indicated the need to be helped to process these feelings and requested a forum in which to do so. Reflective groups run in the style of Balint groups are held for Year 5 and 6 students and are currently being evaluated as a research project. Ideally, individually assigned clinical mentors would be valuable, but this has not proved to be logistically possible as yet.
Second, in 2015, further training of assessors has been implemented to aid feedback consistency. The latter has also been enhanced by the use of Turnitin for marking, with preloaded feedback comments that can be dragged and dropped onto the portfolio document, along with free text as required. Friedman Ben David et al 2 note that “Misclassification into pass/fail or referred categories due to raters’ source of error is a main concern regarding disagreement among raters” (p. 544). The use of a rubric with global criteria, 7 a small group of assessors, and moderation of all portfolios close to decision cut points by the PPS coordinators ameliorates this problem, but the latter places a high workload on the coordinators at the end of the academic year. Multifacet Rasch measurement modeling that will take into account interrater differences is planned with one cohort of portfolios at the end of 2015.
Ongoing evaluation will be continued through student feedback. However, we see the need for a larger independent evaluation once the first students introduced to the portfolio have reached the end of their undergraduate training (end of 2017). In addition, it may be interesting to investigate student performance in the PPS domain compared with their performance in other domains with a different form of assessment (eg, Applied Science for Medicine), as suggested by Friedman Ben David et al. 2 We anticipate that there is unlikely to be an exact correlation, given that it assesses a different form of knowledge and skills.
Conclusion
The implementation of a portfolio for the assessment of the PPS domain in the medical program at the University of Auckland has proved to be an important part of the redeveloped curriculum. It has highlighted the importance of PPS topics and has provided an opportunity and incentive for students to reflect on aspects of their personal and professional development, with the aim of improving their practice as doctors. In the process, we have learned several aspects about students that previously may have remained unknown, which has led in some cases to conversations with students that will hopefully make a difference to their learning and development trajectories. We have also seen many high-quality and insightful portfolios that demonstrate a synthesis of personal and professional qualities that are inspiring and which we hope are an indication that these students will develop into excellent doctors. With the increasing use of, and requirement for, the compilation of professional portfolios in the medical profession internationally, especially at postgraduate level, there is real benefit for students in becoming familiar with this process of pregraduation.
The portfolio implementation process has also provided challenges, particularly with respect to student perception of the relevance of reflective practice in the
It has also been a learning process for the domain coordinators introducing the assessment. In particular, we learned the value of a cyclic, flexible, and ongoing process of evaluation that has allowed us to continuously reflect on what is working and what needs to change in order to be responsive to student needs and at the same time create a curriculum change that aims to create better doctors. As stated by a Year 5 student, “I really appreciate that the PPS department seems to take real interest in student feedback and changes their methods based on that–-thank you!” We hope that as students come to trust our responsiveness, they will become increasingly open to the importance of taking on the responsibility for their own learning in the PPS domain.
Author Contributions
Analyzed the data: JY and FM. Wrote the first draft of the manuscript: JY. Contributed to the writing of the manuscript: JY and FM. Agree with manuscript results and conclusions: JY and FM. Jointly developed the structure and arguments for the paper: JY and FM. Made critical revisions and approved final version: JY and FM. All authors reviewed and approved of the final manuscript.