Centre for Engineering Education Blog

Tag: ChatGPT

  • Attitudes Towards Generative AI: A Study with First-Year Computer Science Students

    Varvara Papazoglou is a Teaching Associate in the School of Computer Science, University of Sheffield. In this post Varvara shares the work she led to investigate student perceptions of AI that found growing confidence, excitement and acceptance. The work was presented at the 18th annual International Conference of Education, Research and Innovation (ICERI 2025).

    The study started during the academic year 2023-24 and was intended to track how the perceptions, expectations, and technical understanding of our undergraduate students shifted as they gained direct experience with GenAI technologies. With a team that included Alex Lucas and Prof. Rob Gaizauskas, we redesigned the first two lab sessions for a first-year module called Machines and Intelligence so that they included a practical introduction to Generative Artificial Intelligence (GenAI). Our study spanned two academic years, and tracked how the perceptions, expectations, and technical understanding of students shifted as they gained experience with GenAI technologies . To measure this, we invited volunteer participants from both cohorts to complete two online surveys, one before and one after engaging with the GenAI-focused curriculum of the module.

    Students on laptops in a classroom setting

    These questionnaires were administered three to four weeks apart, using unique IDs to match participant responses while preserving their anonymity. Structurally, they consisted of 13 core questions common to both iterations. Questions identified the specific GenAI tools students utilised and their typical purposes for doing so with, and 5-point Likert scales were used to evaluate viewpoints on GenAI’s performance, the identification of AI-generated content, and general feelings of excitement or concern. The perceived impact of AI on future careers, potential job obsolescence, and the ethics of GenAI use in university and workplace settings were explored. To ensure a comprehensive analysis, most of these questions included non-mandatory open-ended sections for qualitative insights, concluding with a final opportunity for participants to offer additional comments.

    Testing Turing

    The first lab focused on a practical adaptation of the Turing Test.

    Turing Test Diaghram

    Students were split into groups of 6-7, paired with another group to play the roles of “interrogator” (C in Turing’s original formulation) and “interrogated” (both A (the man) and B (the woman/machine)) interchangeably.

    To maintain anonymity and remove visual cues, all communication happened through a shared online document. The “interrogated” groups were tasked with choosing whether to provide the answers of an actual team member or to use ChatGPT or Gemini to produce human-like responses. The latter required students to act as prompt mediators; for example, they might instruct the model to adopt the persona of a 20-year-old first-year computer science student while keeping answers within a specific word count. This exercise forced students to grapple with the actual capabilities and flaws of these models. For their assessment, students submitted reflections on the following:

    ●      As interrogators, whether they successfully identified the machine and what specific linguistic cues or mistakes led to that conclusion.

    ●      As the interrogated group, whether they successfully deceived the interrogators into making the wrong identification, and whether this success was attributed to the AI’s capabilities or their own careful prompt engineering or other factors.

    ●      Whether the Turing Test remains a valid benchmark for machine intelligence, based on their experience with the given task.

    The second lab shifted towards the ethical and practical implications of AI in Higher Education. After reviewing online University resources regarding guidelines and policy on the use of GenAI, students analysed and critically evaluated 10 short descriptions of use cases on GenAI in Higher Education, split into two groups of five, involving use by educators and five use by students, ranging from AI-generated lecture content to code authoring with GenAI.

    Groups debated the advantages and disadvantages of these scenarios, helping them develop an informed, critical perspective on how to use these tools responsibly throughout their degree.

    Confidence, Excitement and Acceptance

    Students working on a drone

    To understand how these labs changed perspectives, we conducted pre- and post-task surveys, analysing 33 matched responses from the 2023-24 and 2024-25 cohorts. The results revealed the following in student literacy:

    ●      Initially, students were fairly confident that GenAI could handle complex tasks. After the labs, this confidence actually dropped (from an average of 3.8 to 3.5 on a 5-point scale), suggesting that students gained a first-hand understanding of the inherent “hallucinations” and limitations.

    ●      Student confidence in their ability to identify AI-generated content (based on cues such as unnatural or “over-polished” text) rose slightly, from an average of 3.2 to 3.5. Overall, as indicated by their responses to our survey questions, most participants had some experience and a good understanding of cues that can reveal GenAI output before our teaching interventions. However, among participants with no prior experience, greater confidence in recognising AI-generated material was observed.

    ●      Interestingly, despite becoming more aware of the risks, students’ willingness to use GenAI in their studies increased (from 3.7 to 4.3). In addition, in their comments, students emphasised that, because GenAI tools and technologies are here to stay, the priority must be acceptance and education, so people are aware of how to use them effectively to maximise utility while minimising risk.

    ●      While the media often portrays AI as a threat to jobs, our students largely viewed it as a transformation. Many saw the obsolescence of certain tasks not as a negative, but as an opportunity for new roles and increased productivity. Interestingly, students expressed similar feelings of excitement and acceptance for GenAI use in both Higher Education and the workplace.

    Computer image of a face with letters coming out of it
    Conclusion

    Our findings suggest that early, structured engagement with GenAI, tied to subject-specific material, can foster more balanced and informed perspectives among students. Students with more awareness of risks and limitations seem to be more optimistic of the impact and potential of GenAI. However, we are mindful of the modest scope of our current research. Given the relatively small sample size and the specific context of our introductory module, these results serve as preliminary insights rather than definitive conclusions. We have identified areas for improvement, particularly in deepening students’ awareness of ethical and societal risks while reducing the likelihood that students feel overly confident about their ability to identify and manage AI-generated content without necessarily being aware of the full range of risks.

    Varvara Papazoglou is a Teaching Associate in the School of Computer Science, University of Sheffield. Her research interests include Artificial Intelligence, Music Information Retrieval, Computational Linguistics and Natural Language Processing, as well as Educational Research in Computer Science.

    When citing this work, please use the following citation:
    Papazoglou, V. (2026). “Attitudes Towards Generative AI: A Study with First-Year Computer Science Students” Centre for Engineering Education Blog, The University of Sheffield, Sheffield, UK. June 2026.     https://ceesheffield.co.uk/2026/06/09/attitudes-towards-generative-ai-a-study-with-first-year-computer-science-students/

  • Becoming Comfortable with Black Box Engineering

    In this article, Professor Andrew Garrard from the University of Sheffield and Dr Parakram Pyakurel from New Model Institute for Technology and Engineering discuss whether engineering education’s traditional resistance to “black box” thinking remains fit for purpose. As simulation software, Industry 4.0 methods, and generative AI make it harder for engineers to understand the tools they use, is it time to take a pragmatic approach in education and consider black box engineering not as a threat to be resisted, but as an inevitability to be acknowledged?

    cartoon of modern engineering in a nutshell

    Cartoon images generated by AI using the prompt “Please can you read this article and come up with a simple pencil drawing cartoon that makes light of the general themes of the piece?”. This image was created using Open AI’s ChatGPT 5.4.

    There is a widely held belief amongst engineering educators that students should be discouraged from relying on a “black box” approach to engineering; that is, producing results using methods and tools with little or no understanding of how they work. Moreover, it can be considered dangerous to certify a graduate as competent if they accept, on faith, the validity of results generated by means they do not understand.

    To what extent, however, is this view an anachronism, tightly clung to by academia but no longer reflective of real-world engineering practice?

    Cartoon images generated by AI using the prompt “Please can you read this article and come up with a simple pencil drawing cartoon that makes light of the general themes of the piece?”. This image was created using Anthropic’s Opus 4.6.

    Consider, for example, predicting how a system will behave through engineering simulation, such as by using the finite element method. In previous generations, it was conceivable that engineers could build, from the ground up, the software required to perform finite element calculations. This was possible because the limits of computational power meant that only relatively simple systems could be analysed using comparatively straightforward numerical techniques.

    Fast forward to today, and modern engineering simulation software can consist of hundreds of gigabytes of code and mind-bogglingly complex algorithms designed to optimise every stage of the process. Take the generation of computational meshes. In the past, engineers constructed meshes manually, deciding the position of nodes and connecting them together with lines. Today, powerful automated meshing algorithms generate highly complex meshes as part of a largely turnkey process.

    It is unrealistic to expect engineering students, who are learning multiple topics across a wide range of domains, to understand the detailed inner workings of every piece of software they use. As the ability to understand precisely what engineering simulation software is doing has diminished, it has been replaced by an increased emphasis on heuristics: I chose this turbulence model because experience suggests it performs well for this type of fluid flow.

    Cartoon depicting black box engineering

    Cartoon images generated by AI using the prompt “Please can you read this article and come up with a simple pencil drawing cartoon that makes light of the general themes of the piece?”. This image was created using Google’s Gemini 3.1

    The same “black box” approach underpins many aspects of Industry 4.0, and more recently Industry 5.0. Rather than fully understanding the physical mechanisms governing the dynamics of engineering systems, sufficiently large datasets can be used to predict outcomes accurately through sophisticated statistical methods.

    The world is already full of black-box engineering, and this trend is only likely to accelerate with the rapid advancement of artificial intelligence. While it may be extremely challenging to fully understand all the methods used in modern, complex engineering software, the algorithms involved are at least deterministic. By contrast, the stochastic outputs of Generative AI, like large language models, means that determining how they will respond to any given input is not something anyone can reliably predict, even those who have built the models. As AI becomes increasingly embedded within software products, additional opaque layers will accumulate, making it ever more difficult to truly understand what the computer is doing.

    Whether the adoption of a “black box” approach to engineering is desirable remains open to debate. What seems less debatable, however, is its inevitability in the modern world. We can either acknowledge this reality, or pretend it is not happening.

    Once we admit this – what are the implications?

    The increasing complexity of computational methods, together with the advent of AI systems whose behaviour can be difficult to fully understand, may appear to be a distinctly modern problem. However, making predictions about how systems will behave with limited understanding of the underlying governing principles has long been the standard approach of empiricists.

    Often attributed to Heisenberg is the quote:

    “When I meet God, I am going to ask him two questions: Why relativity? And why turbulence? I really believe he will have an answer for the first.”

    As a species, we remain unable to analytically calculate turbulent flow. Yet engineers are able to model aeroplanes, pipelines, and wind turbines with impressively high precision. While imparting conceptual understanding of fundamental engineering principles is important, we may be overlooking opportunities to train students in data analysis, interpretation, and subsequent processing.

    As educators, our task is to equip students with the understanding and tools necessary to have a positive impact on the world. In already crowded engineering curricula, content must compete for inclusion among the many topics considered essential for an undergraduate programme. If complexity, big data, and stochastic computation are becoming commonplace, is there a need to shift some emphasis away from traditional physical sciences toward statistics, empiricism, and validation?

    When citing this work, please use the following citation:
    Garrard, A & Pyakurel, P. (2026). “Becoming comfortable with black box engineering”. Centre for Engineering Education Blog, The University of Sheffield, Sheffield, UK. March 2026. https://ceesheffield.co.uk/2026/10/27/blackboxengineering/