This study explores potential disparities between flight instructor evaluations and pilot self-assessments in the context of full flight simulator training. Evaluated performance was based on the Competency-based Training and Assessment framework, a recent development of competency-based education within aviation. Self-assessed performance is derived from survey responses and debriefing interviews. The simulator session involves eight multi-crew pilot training graduates and eight experienced flight captains, encompassing two tasks featuring sudden technical malfucntions during flight. The flight instructor’s evaluations reveal no significant differences in pilot performance. However, disparities become apparent when pilots engaged in reflecting their performance. Novice pilots, despite perceiving both tasks as easy, exhibited an overconfidence that led them to underestimate the inherent risks. Conversely, experienced pilots demonstrated greater caution towards the risks and engaged in discussing possible hazards. Furthermore, this study highlights the challenge of designing flight simulator training that incorporates surprise elements. Pilots tend to anticipate anomalies more readily in simulator training than during actual flights. Thus, this study underscores the importance of examining how pilots reflect on their performance, complementing the assessment of observable indicators and predefined competencies.