Rethinking the comprehensive test on qualitative reasoning for designers

A battery of tests assessing the cognitive skills needed for the conceptual design is being developed. Tests on Divergent thinking and visual thinking are fully developed and validated. The first version of the qualitative reasoning test has also been developed; this paper focuses on the lessons learned from testing of the first version of the test (alpha version) and the improvements made to it since then. A number of problems were developed for each indicator of the qualitative reasoning skill (deductive reasoning, inductive reasoning, analogical reasoning, and abductive reasoning). Later, a protocol study was done with the problems to make sure that the problems assess the desired skills. The problems were also given to a randomly chosen population of undergraduate senior-level or graduate-level engineering students. Data was collected from the test results on the possible correlations between the problems (e.g. technical and non-technical problems); feedback on clarity, time allocation, and difficulty for each problem was also collected. Based on all of the observed correlations, the average performance of the test takers, and test parameters such as validity, reliability, etc. the beta version of the test is constructed.