Radioprotection 2025, 60( 3), 211 – 220 © S. Ito et al., Published by EDP Sciences 2025 https:// doi. org / 10.1051 / radiopro / 2024056
Available online at: www. radioprotection. org
ARTICLE
Producing nuclear disaster prevention materials with artificial intelligence chatbots: comparison of ChatGPT-3.5, Copilot, and Gemini output with google search results
S. Ito 1,*, E. Furukawa 2, T. Okuhara 2, 3, H. Okada 2, 3 and T. Kiuchi 2, 3
1 School of Nursing, Kitasato University, 1-15-1, Kitasato, Minami-ku, Sagamihara-city, Kanagawa 252-0373, Japan. 2 University hospital Medical Information Network( UMIN) Center, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo
113-8655, Japan. 3 Department of Health Communication, School of Public Health, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo,
Bunkyo-ku, Tokyo 113-8655, Japan. Received: 19 August 2024 / Accepted: 24 November 2024
Abstract – Objective: To compare the understandability, actionability, and readability of AI chatbotgenerated text and webpage text about nuclear disasters. Methods: In this cross-sectional study, we compared the understandability, actionability, and readability of ChatGPT-3.5, Copilot, and Gemini generated texts and web page sentences about radiation. The keywords related to radiation were extracted using Google Trends. A Google search was performed using the extracted keywords, and the top 8 pages for each keyword were extracted. The AI chatbot generated two types of sentences: normal level and 6th grade level. The Japanese version of the Patient Education Materials Assessment Tool( PEMAT-P) was used to rate the understandability and actionability of each text. The higher the score, the higher the perceived ease of understandability and actionability and the cutoff for both was set at 70 %. The jReadability was used to quantitatively assess the readability of Japanese texts. Results: With regard to understandability, Copilot( n = 22, 71.0 %) and Gemini( n = 26, 92.9 %) 6th grade level texts had significantly higher percentages of 70 or higher, while Google Search had a significantly lower percentage of 70 or higher( n = 58, 32.8 %; p <. 05). Gemini at the normal level( n = 69, 55.2 %) and Copilot( n = 74, 55.6 %) and Gemini( n = 73, 56.2 %) at the 6th grade level had significantly higher percentages of very readable to somewhat difficult responses( p <. 05). Conclusions: The Japanese sentences generated by the AI chatbot were easier to read than the Google search results; the prompt of 6th grade level improved the readability of Japanese sentences. Thus, the AI chatbot can be an effective tool to promote understanding of radiation disaster prevention.
Keywords: nuclear power / patient education / ChatGPT / Copilot / Gemini
1 Introduction
On March 11, 2011, many people were exposed to radiation as a result of the accident at TEPCO’ s Fukushima Daiichi Nuclear Power Station( Tsubokura et al., 2012; United Nations Scientific Committee on the Effects of Atomic Radiation, 2015). At the time of the nuclear accident, information related to radiation and radioactivity was scarce, and explanations by experts and others were difficult for the public to understand, leading to anxiety and emotional distress among local residents( Rubin et al., 2012). In Japan today, nuclear disaster prevention pamphlets for local residents have been prepared in each region( Cabinet Office and Fire and
* Corresponding author: shinyai @ nrs. kitasato-u. ac. jp
Disaster Management Agency, 2013). Ito et al. evaluated the quality of Japanese language nuclear emergency preparedness pamphlets available free of charge online and reported that 61.2 % of the pamphlets were easy to understand and 49.1 % were easy to act upon according to the Patient Education Materials Assessment Tool( PEMAT)( Ito et al., in press). However, they reported that 88.8 % of these nuclear disaster preparedness pamphlets were written in a manner that required reading comprehension skills capable of comprehending of substantially technical text. Given that the Internet is reported to be a useful method of distributing information to the general public in emergency situations such as nuclear disasters( Kanda et al., 2014), it is important to create nuclear disaster preparedness information that is easily understood by local residents( Gauntlett et al., 2019; Goto et al., 2018; Hellier et al., 2014; Ito et al., 2017; Ohno and Endo, 2015).
This is an Open Access article distributed under the terms of the Creative Commons Attribution License( https:// creativecommons. org / licenses / by / 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.