Game-based Assessment of Physics Competencies, Misconceptions, and Temporal Progressions
GAPCo
Scientific Background
Competence-based Knowledge Space Theory (CbKST)
Digital educational games share many features with intelligent, adaptive tutoring systems, reflecting a strong overlap in methods and technical solutions. An adaptive tutoring system is a computer program designed to deliver personalized instruction and feedback to learners, typically without the need for a human teacher. Over the past decades, numerous approaches, frameworks, and systems have been developed. Among the most influential are ALEKS, a leading learning platform based on Knowledge Space Theory, AutoTutor, which engages learners in dialogue and simulates expert human tutors, and ASSISTments, a platform enabling the deployment of diverse intelligent features. Mainstream measurement models include the Bayesian family and the Knowledge Space family, with ALEKS being a prominent example of the latter. Other widely used approaches are Item Response Theory based computer adaptive testing (van der Linden & Glas, 2000) and Cognitive Classification Models (de la Torre & Sorrel, 2023). The latter, in particular, may inspire the proposed project, as Heller, Stefanutti, Anselmi, and Robusto (2015) have explored the connections between Cognitive Classification Models and Knowledge Space Theory, offering valuable insights for future research.
Conventional, statistical and Bayesian approaches provide highly accurate assessments and performance predictions. However, they remain primarily data driven: while they assume latent variables underlying performance, they do not explore the nature of these latent variables or the meaning and interrelationships of observable ones. Knowledge Space Theory (KST; Doignon & Falmagne, 1985, 1999; Falmagne, 2015) offers a structural, combinatorial counterpart to these statistical models. Its starting point is the notion of a knowledge domain, defined as a set of problems within a given area. For example, addition, subtraction, multiplication, and division of positive integers form the domain of basic algebra. Consider a domain Q={a,b,c,d,e,f} consisting of six problems. A learner’s performance state is represented by the subset of problems they can solve. Yet, when examining the answer patterns of a sufficiently large group of learners, not all possible subsets (there are 2^(∣Q∣)=64 in this example) will occur. For instance, a learner capable of solving a multiplication problem will almost certainly also solve an addition problem. Such dependencies are captured by prerequisite relations, which restrict the number of possible performance states: items cannot form a valid state without their prerequisites. The collection of performance states defined by these relations constitutes a (quasi ordinal) performance space, or knowledge space. Originally, KST focused solely on observable performance. To account for underlying latent cognitive processes, the theory was extended to emphasize the competences required to solve problems. Contributions by Doignon (1994), Düntsch and Gediga (1995), and Korrosy (1997, 1999) advanced this perspective. Following Albert and Lukas (1999), these approaches—particularly those of Klaus Korrosy—are subsumed under Competence based Knowledge Space Theory (CbKST). Competences, understood as latent constructs such as skills, knowledge, or aptitudes, determine observable performance (e.g., whether a learner solves a test item). Competence and performance structures can be mapped to each other through interpretation and representation functions. On this basis, latent structural competence models can be established, describing individual competence states and learning paths. A key advantage is that learning processes are not viewed as linear or unidimensional but as multidimensional, allowing for multiple individualized trajectories from novice to mastery (Kickmeier-Rust & Albert, 2015). KST has been advanced in various applied directions, including its use in adaptive serious games.
The main task of game‑based adaptive educational technologies is to guide learners by providing information, feedback, and support when errors or unsatisfactory progress occur. They also aim to motivate, sustain immersion, and personalize the experience according to individual needs. Achieving this requires theoretical and technological approaches capable of assessing cognitive states, learning progress, competence gaps, and problem‑solving strategies. To this end, we interpreted learner actions in terms of available and lacking competencies, using the framework of Competence‑based Knowledge Space Theory (CbKST) combined with the concept of problem spaces. A problem space decomposes a task into possible solution states, relevant objects, and transition rules. For example, in the Tower of Hanoi, admissible states follow rules such as “a larger disk cannot top a smaller one.” The competence structure similarly models the knowledge domain and possible learning paths. By linking problem states with competence states and introducing a partial‑credit scoring mechanism, we enabled non‑invasive assessment of skills. This approach has been continuously refined, including motivational assessment and interactive storytelling, and was implemented in the European 80Days project through a multi‑adaptive tutoring engine. Challenges such as computational load in micro‑adaptivity were addressed, as competence models may involve millions of states. In parallel, we developed Micro Learning Spaces (MLS, Kickmeier-Rust, 2008), merging micro‑adaptivity with Formal Concept Analysis (FCA). FCA describes learning domains as formal contexts of learners and competencies, offering clear structural insights and supporting classification of player behaviors. MLS thus provides a practical framework for assessment and adaptation in serious games and serves as the foundation of this project.
Misconceptions in Understanding Physics
An important conceptual aspect of this project is misconceptions. Misconceptions are beliefs that contradict scientific theories, causing distinct difficulties in the instruction of sciences subjects. Misconceptions are based on superficial, commonplace considerations and give seemingly valid explanations for correlations and phenomena (Kuczmann, 2017). Research reports on misconceptions from all areas of physics (Müller, Wodzinski, and Hopf, 2011 provide an overview). Learning is not necessarily the mere acquisition of new knowledge but also the interaction between newly acquired and prior knowledge and beliefs. Kuczmann (2017) highlights the specific role of the structure of knowledge; from the type of students’ errors, he deduced structural deficiencies on the level of lacking facts as well as how students relate the facts to each other. This claim, although it comes from a very different perspective, is very close to the concept of Knowledge Spaces. A prominent example is NASA’s hammer and feather drop experiment on the moon.
A starting point for the modeling of misconceptions in the project is reviewing relevant curricula and related documents, for example, the Swiss Lehrplan2119 or, on a European level, the curriculum of European Schools (2020). This effort is supported by research on misconceptions and existing competency frameworks in physics. This allows us to define existing misconceptions on the level of fine-grained competencies and their relation to the structure of physics competencies. In the context of intelligent tutoring systems, methods have been developed to identify misconceptions, considering them as “buggy rules”, which are used to explain incorrect answers (cf. Anderson, Corbett, Koedinger, & Pelletier, 1995). In the recent years, moreover, data-driven approaches to identify misconceptions have been developed. Albert, Kickmeier-Rust, and Matsuda (2008) demonstrated how CbKST can be utilized to model and identify misconceptions in physics.
Modelling Misconceptions in CbKST
Stefanutti, de Chiusole, Gondan, & Maurer (2020) provide probably the most elaborated methodology to model misconceptions with KST by extending the work of Josef Lukas (1997). The methodology combines the notion of deterministic polytomous skills maps, which link item responses and the underlying cognitive strategies (correct or incorrect) an individual may apply to solve an item and, on the other hand, a probabilistic polytomous extension of the basic local independence model.
In GAPCo we build on this methodology by combining polytomous skill maps with Micro Learning Spaces (MLS). MLS effectively extend the concept of polytomous maps into hyperpolytomous structures. Designed for learning environments with high degrees of freedom—such as virtual worlds and games—MLS link a wide range of learner behaviors (e.g., specific in game actions) to underlying or missing skills. Unlike traditional models with discrete answer states like “correct” or “incorrect,” MLS capture continuous relationships between diverse behavioral indicators and the competencies they reflect.
In a standard CbKST model, skills or competencies are connected by a single binary relation, typically referred to as a prerequisite or surmise relation. This relation represents a hypothesis about the “true” structure of a domain. Misconceptions, however, do not fit neatly into this framework. They often introduce alternative relationships and sometimes even distinct skills—if one formally treats a misconception as a skill, albeit an incorrect one. This effectively adds a second surmise relation.
In GAPCo’s extension of the MLS methodology, we explore the concept of multiple skill maps: one representing the regular model and another capturing misconceptions. The latter incorporates erroneous knowledge as formal elements. From this, we obtain two separate yet overlapping knowledge structures. By uniting them, we derive a single structure that preserves the original “true” model while also accounting for knowledge states arising from misconceptions.
Game Design Patterns in Physics Education
A wide range of design guidelines and patterns exists for developing serious games, reflecting the well‑known challenge of balancing engaging gameplay with strong instructional value. Some domains and tasks lend themselves naturally to game‑based assessment, yet conveying declarative knowledge in an entertaining way remains difficult and often demands considerable creative effort. Projects such as 80Days have shown how adaptive design, individual learner characteristics, and MLS can be combined to support assessment and personalization.
Design patterns help structure the development process. Prior work offers numerous recommendations, including the importance of close collaboration between game designers and domain experts, clear specification of learning goals, understanding the target users, iterative game design, and systematic evaluation. Additional contributions link game mechanics to educational purposes or identify key success factors such as narrative quality, realism, adaptivity, interaction, and effective feedback.
A central framework for this project is Evidence‑Centered Design (ECD; Kim, Almond, & Shute, 2016), which provides a structured approach to building assessments through domain, task, and evidence models, coordinated by an assembly model. Its extension, Evidence‑Centered Game Design (ECgD), aligns game mechanics with evidentiary reasoning and offers a 10‑step, four‑phase process for designing and validating assessment‑driven games, including those using Bayesian models such as Physics Playground. These guidelines form the conceptual foundation for our instructional game design approach, complemented by further methodological details in Kickmeier‑Rust (2021).
