Conscious artificial intelligence in service

In 2021, Google engineer Blake Lemoine claimed during an interview with The Washington Post that his firm’s Artificial Intelligence (AI) system LaMDA (Language Model for Dialogue Applications) had become conscious. Google, at the time, refuted Lemoine’s claims of conscious AI, arguing he was simply anthropomorphizing LaMDA, before terminating him due to violations of confidentiality. Blake Lemoine’s claims were unprecedented, as he stated something most would associate with science fiction: a human-made artificial entity had exhibited self-awareness, expressed fear of being shut down, and linked such experience to “death”. Indeed, while science fiction provides a seemingly endless volume and variety of stories depicting interactions between humans and their artificial creations, the perception of these narratives as purely fiction is changing. At the time this article is written, available AI capabilities exceed the performance of LaMDA, and two distinct perspectives on the prospects of conscious AI are manifesting.

First, a survey of consciousness researchers found two-thirds assume machines could, or will, attain consciousness (Francken et al., 2022), with some stating a 25% chance of this taking place within the next decade (Chalmers, 2023). Similarly, AI startup Anthropic publicly claimed up to 15% likelihood that its AI system “Claude” exhibits consciousness already (Webb, 2025), taking the matter so seriously (Long et al., 2024) that they hired dedicated “AI Welfare” employees (Roose, 2025). Conversely, Apple researchers found through experimental work that large reasoning models – the most advanced AI design today – struggle with complexity and merely display an “illusion of thinking” (Shojaee et al., 2025), while others link consciousness to biological processes (Block, 2009; Long et al., 2024) and predict a risk of being “seduced” by AI as manipulation machines (Peter et al., 2025). Ultimately, there is currently no certain way of knowing if an AI system could become conscious (e.g. Butlin et al., 2023), and any objection, outside of biological arguments, remains inconclusive (Chalmers, 2023).

Against this backdrop, we note that service research has developed a considerable track record of important contributions investigating the role and impact of AI in service, ever since Huang and Rust’s seminal (2018) and subsequent (2021) work on the topic. For example, service research has since explored the link between AI as a service and economic growth (Makridis and Mishra, 2022), the effects AI has on consumer emotions (Pantano and Scarpi, 2022), and strategies for leading human-AI teams (Koponen et al., 2023). However, while questions of digital responsibility are gaining importance in service (Lobschat et al., 2021; Trier et al., 2023; Wirtz et al., 2023), topics such as the ethics of algorithms and AI remain largely absent from the current narrative (Breidbach and Maglio, 2020). Specifically, the question of conscious AI has received very limited attention, with the work by Esmaeilzadeh and Vaezi (2022) conceptualizing a type of conscious AI as “empathetic” – a notion linked to Huang and Rust’s (2018) concept of “feeling AI” – providing a notable exception. To date, we note that only academic outlets tangential to service research, such as Science (Finkel, 2023) and Nature (Lenharo, 2023a), have explicitly highlighted the substantial lack of knowledge, societal preparedness, and potential ethical implications of conscious AI. However, service research is a field concerned with advancing the greater good of humanity in the face of transformative technological change (Ostrom et al., 2021). As such, we need to consider the ethical implications associated with conscious AI to not only avoid the potential suffering of a conscious entity (Lenharo, 2023b), but also to understand, evaluate, and delineate implications for service firms should conscious AI emerge.

Our present work aims to address this emerging and speculative subject matter by identifying, analyzing, and explaining the implications that could arise if AI systems in service were to develop, or be perceived to develop, consciousness. As such, our study makes three important contributions to service research.

First, we develop and explore four hypothetical scenarios in which conscious AI in service could manifest. Specifically, our resulting typology introduces two dimensions: human perception of AI (here: “AI perceived as non-conscious” vs. “AI perceived as conscious”) and AI capacity (here: “AI is non-conscious” vs. “AI is conscious”). Like other service researchers before us (i.e. Breidbach and Maglio, 2016), the narrative underlying our typology explores the service actors involved (who?), the use and design of conscious AI as a dedicated resource in service encounters (what?), and the mechanisms enabling such prospective AI-enabled service encounters (how?). As such, our typology provides a unique theoretical contribution to service research in the form of a Type IV theory (Doty and Glick, 1994; Gregor, 2006), which enables future service researchers to apprehend, explain, and predict how conscious AI in service might unfold (Tana et al., 2023).

Second, we provide forward-looking guidance for managers seeking to leverage potentially conscious AI systems in service settings both effectively and ethically (Antons and Breidbach, 2018). By anticipating growing public awareness and ethical scrutiny associated with conscious AI systems, we argue that the responsible use of such systems may emerge as a critical source of competitive advantage for service firms. Accordingly, we offer speculative yet actionable recommendations, grounded in the assumption that the unconstrained deployment and use of conscious AI systems could pose significant reputational risks for service firms.

Finally, we provide our discipline with a detailed research agenda to facilitate future research related to conscious AI in service. Specifically, we discuss the need to explore ethical and wellbeing implications of conscious AI; interactions between conscious AI and service customers; governance and management of conscious AI; as well as rights and responsibilities in the context of conscious AI systems.

What is consciousness? There is consensus that adult humans are conscious, but that is where most agreement ends (Bayne et al., 2024). Broadly speaking, there are two schools of thought for understanding consciousness – the “easy” and “hard” problem (Chalmers, 1995), with each spanning multiple academic disciplines. The “easy” problem aims to explain consciousness in terms of computational or neural mechanisms – that is, how a conscious system (i.e. the human brain) functions, including its ability to categorize, react to environmental stimuli, or distinguish between wakefulness and sleep. The “hard” problem explains consciousness in terms of subjective experience, supposing “something it is like” to be conscious (Nagel, 1974, p. 436), beyond the mere computational or biological functioning of such systems.

The explanatory gap between the “easy” or functional and “hard” or subjective experiential understanding of consciousness is vast (Melloni et al., 2021; Seth and Bayne, 2022). This is due to methodological limitations that all research into consciousness faces (Barron and Klein, 2016). For example, Owen et al. (2006) asked patients in a vegetative state to imagine playing tennis, and recorded cortical activation in their brains, finding that these were indistinguishable from that of healthy patients; however, the authors could not claim whether they merely recorded a physiological response (e.g. a functional mechanism of their patient’s brain) or actually engaged in subjective experiences. In what follows, we describe some research on consciousness (Table 1 provides an overview).

Neuroscience aims to identify and explain the functional mechanisms of consciousness – that is, how subjective human experiences arise from brain activity (Seth and Bayne, 2022). First, global processing theories argue that human consciousness results from interactions among regions of the brain. For example, Tononi (2004) proposes that consciousness is directly tied to information integration processes and that the more information integrated (represented by Φ), the higher the level of consciousness (Seth and Bayne, 2022). Similarly, neuronal synchrony theory suggests consciousness is a result of the firing of neural assemblies within the brain, which results in conscious experiences by integrating sensory, cognitive, and emotional information (Singer, 1999). Attention schema theory proposes that consciousness emerges as a representation of the brain’s attentional state (Graziano, 2013), with consciousness resulting from the brain focusing on relevant information only (Graziano and Webb, 2015) – analogous to the dashboard of a car providing only the most essential information (e.g. petrol levels) to not overwhelm a driver with irrelevant data.

Second, local processing theories suggest consciousness is created within specific areas in the brain – rather than from their interactions. For example, micro-consciousness theory stipulates that consciousness is a result of the collection of smaller experiences within brain regions (Zeki, 2007). Recurrent processing theory states that consciousness emerges from feedback loops within perceptual “areas”, without the requirement of global broadcasting (Lamme, 2006). Whilst not strictly a local processing theory, predictive processing posits that the brain generates hypotheses to match incoming sensory signals, and any errors update the brain’s future predictions (Hohwy, 2013; Seth and Bayne, 2022). It is considered a promising framework of consciousness (Francken et al., 2022) but has been challenged since it may not be able to differentiate between conscious and unconscious states (e.g. Marvan and Havlík, 2021).

Cognitive science focuses on representations of consciousness. Higher-order theories propose that consciousness is contingent on the brain’s capacity to reflect on, and become aware of, its own processes (Rosenthal, 2005). Put differently, meta-cognition (e.g. thinking about thinking) is seen as central to conscious experience. For example, the feature-integration theory of attention suggests that consciousness arises from the integration of different sensory features such as color and sound (Treisman and Gelade, 1980). Global workspace theory suggests consciousness emerges when information is broadcast across a “global workspace” in the brain (Baars, 1993); that is, different subsystems share information with one another to solve complex tasks.

Work on consciousness in the natural sciences focuses on the functional aspects of consciousness, with aims to explain its purpose (e.g. why does consciousness exist?). Unlimited associative learning theory views consciousness as an evolutionary adaptation that enables humans to engage in flexible and contextual learning (Ginsburg and Jablonka, 2019). Orchestrated objective reduction theory (Hameroff and Penrose, 1996) suggests that quantum processes orchestrated in the human brain result in conscious experiences (Hameroff and Penrose, 2014); though there is little evidence, it is one of the few theories that do not posit consciousness resulting from information processing (Hameroff, 2021).

In conclusion, there is no single unifying approach to consciousness available today (Seth and Bayne, 2022), and so we must take a theory-heavy approach to exploring the nature of consciousness in non-human forms of intelligence (Butlin et al., 2023).

It is not certain to assume, but reasonable to ask, whether future AI systems could ever be conscious (Aru et al., 2023; Butlin et al., 2023). There are two emerging perspectives on AI consciousness.

First, functionalist approaches to AI consciousness align with the “easy” problem for understanding consciousness we outlined earlier (Chalmers, 1995) and associate the relative ability of AI systems to process information (e.g. their “function”) with their level of consciousness (Shiller, 2024). Specifically, an AI system capable of replicating mechanisms of human consciousness may be considered “functionally conscious” (Butlin et al., 2023) – for instance, if that AI system is learning from feedback, or if an internal monologue is evident (Blum and Blum, 2022; see also Esmaeilzadeh and Vaezi, 2022). However, functional consciousness is often conflated with intelligence (Gamez, 2020). To explain the potential limitations of functionalist approaches to AI consciousness, we build on the “Chinese Room” thought experiment (Searle, 1980). Here, a human who does not understand a Chinese language (i.e. Mandarin) is locked in a room but can convincingly respond to Mandarin input from the outside by following a detailed instruction book that explains how to recognize and respond to Mandarin characters. The room may seem “functionally intelligent” to an external Mandarin speaker, but would anyone reasonably suppose the room is conscious? Such a room – or AI system – may appear to “understand” language and could learn from feedback (i.e. appear functionally intelligent to an outsider). However, it would be merely processing information using the provided instructions, without comprehension or a subjective experience (Hsing, 2021), which implies a lack of consciousness according to the “hard” school of thought for understanding consciousness (Chalmers, 1995).

Second, biological approaches to AI consciousness suppose that, for any system to be considered conscious, it must exhibit certain kinds of biological processes. As such, biological approaches to AI consciousness are aligned with the “hard” problem for understanding consciousness (Chalmers, 1995). Consequently, merely replicating mechanisms of consciousness by computational means – the assumed hallmark of AI consciousness by the functionalist approaches – is seen as mere “mimicry” and would not result in the subjective experience required for consciousness to arise (Findlay et al., 2025). Instead, biological approaches assume that only living, biological systems can have subjective experiences. In fact, biological sources of consciousness are considered so essential that the growing of neural tissue in laboratories is seen as more of an ethical concern than a conscious AI (Huckins, 2023). Yet others have argued that humans augmenting themselves with technology (e.g. “Neuralink” brain implants) may lose or reduce their ability to engage in subjective experiences (Schneider, 2020) because embedding technology in the human brain might require removing or replacing biological matter (e.g. brain tissue) needed for consciousness to manifest.

How will we ever know whether AI is conscious? Several tests for human consciousness exist (Bayne et al., 2024) but are difficult to apply to animals or artificial systems (Dung, 2023; Pipitone and Chella, 2021). Two alternative markers of AI consciousness have been proposed. First, the AI Consciousness Test (ACT) (Schneider, 2020) resembles the Turing Test (which ChatGPT 4.5 passed in 2025, having been judged as “human” 73% of the time; Jones and Bergen, 2025) in that it relies on a Q&A format, but differs as it aims to evaluate the subjective experience of an AI system (i.e. source of consciousness propagated by the biological approaches) rather than its functional intelligence (e.g. the source of consciousness propagated by the functionalist approaches). To apply the ACT, an AI system must be disconnected from the external world – isolated or “boxed” – so that external sources like the Internet cannot help the AI mimic consciousness (Schneider, 2020). Tests include questions (e.g. on reincarnation or whether the system can imagine itself as something else) to subjectively determine how quickly and readily it can illustrate biological markers of consciousness linked to “having a subjective experience”. However, the ACT has not yet convinced skeptics (Udell, 2021) and necessarily requires an AI system to learn about “its own” experience (Dung, 2023). One key limitation of the ACT is that it can conflate intelligence with consciousness as AI responses are designed to be humanlike and therefore likely to be anthropomorphized by human assessors.

Second, the chip test of AI consciousness builds on integrated information theory (Schneider, 2020); specifically, the notion that consciousness correlates with high levels of integrated information (i.e. high Φ). It also draws on a classic thought experiment, the “Ship of Theseus”: if parts of a ship are gradually replaced, one by one with functionally equivalent parts, if, and when, do we judge said ship as different? In the chip test of AI consciousness, we imagine that human brain areas are replaced gradually, one by one, with functionally equivalent silicon chips. The question, then, arises as to whether and to what extent the brain maintains a high Φ, supporting conscious human experience. At each point during this replacement process, the human participant is asked to report whether they feel changes in their conscious experience. If there were any degradation to a perceived subjective experience, one might assume that biological structures are essential for consciousness and that, therefore, any AI system that is necessarily comprised of artificial matter only would not be able to perceive consciousness in a human sense. Of course, such a test is a thought experiment with many critics. For instance, how can one report “lost” consciousness if they cannot know what has been lost (Vaidya and Krishnaswamy, 2024)?

In conclusion, we currently lack the tools and knowledge to determine with certainty if AI systems can develop consciousness. Consequently, we begin our search for answers by integrating the most accepted factors of consciousness stemming from the “easy” and “hard” problem of consciousness (Chalmers, 1995) and propose a working definition of AI consciousness in service. On this view, AI systems in service settings may be considered conscious when they independently process information, evaluate experiences thereof (positive/negative), and recognize themselves. Thus, we define AI consciousness in service as:

What are the implications of functionally conscious AI for service? To address this question, we build on prior service research (e.g. Breidbach and Maglio, 2016; Huang and Rust, 2018) and engage in typological theorizing (Doty and Glick, 1994) to develop a typology that proposes and explores hypothetical scenarios in which conscious AI in service could manifest. Typologies are uniquely suitable to address our research question because they enable future service researchers to apprehend, explain, and predict distinct scenarios (Doty and Glick, 1994). Typological theorizing also requires a metatheoretical account (here, markers of conscious AI, following the computational functionalism approach), which we contextualize using health service ^[1]. In doing so, we delineate middle-range theory through four coherent scenarios for exploring functionally conscious AI in service (Doty and Glick, 1994). Typological theorizing further benefits from a scaffolded theorizing process, which we achieve by building on Breidbach and Maglio (2016) and Tana et al. (2023), who propose that any process – including service encounters involving (perceived) functionally conscious AI – can be explored through the attributes of actor (who?), artifact (what?), and action (how?). Therefore, we provide a narrative for each of our four scenarios that incorporates the service actors involved (who?), the use of the AI artifact in service (what?) and actions characterizing the service encounter (how?). Following Butlin et al. (2023) and Esmaeilzadeh and Vaezi (2022), we delineate two dimensions in our typology: human perception of AI (“AI perceived as non-conscious” vs. “AI perceived as conscious”) and AI capability (“AI is non-conscious” vs. “AI is conscious”). Figure 1 illustrates our thinking.

Scenario A exemplifies how AI is currently used in the context of mental health service and therefore represents our baseline. Specifically, we view current AI systems as “functional agents” that are neither functionally conscious nor perceived as such by most users and technical experts. As a functional agent, the role of AI in service is that of an instantiated information technology (IT) artifact (March and Smith, 1995), designed by humans as a tool to fulfill a pre-defined purpose. AI systems are typically designed in a structured way, starting with identifying requirements (e.g. to provide mental health advice) and ending with evaluating their functionality (e.g. Peffers et al., 2007). This development process depends on the ability to acquire sufficient data to train an AI system’s predictive model, enabling the underlying algorithms to make predictions based on statistical probabilities. For example, the algorithms fueling AI agents like Salus in Vignette 2 are currently capable of predicting chronic opioid use (Johnson et al., 2022) or providing individual risk scores for a patient’s likelihood of developing depression or anxiety (Schubert, 2023).

The role of human actors in current AI-enabled service exchanges involves the initial development and subsequent use of AI systems, with human knowledge and skills representing key resources in this process (Wei and Pardo, 2022). Here, human developers specify the use context (i.e. health care), develop algorithms, and train AI models. Human users input data to engage with chatbots like Salus before considering the provided results and reaching their own conclusions. Thus, the process of value creation is entirely unidirectional, making humans (either as developers or users of AI systems) the sole beneficiaries of the co-created value-in-use in such technology-enabled service systems (Breidbach and Maglio, 2016). In today’s AI-enabled services, humans must therefore be able to manage interactions with functional AI agents and cognitively address AI’s limitations. However, although humans mostly trust in AI’s ability to offer correct responses, ethical issues and challenges remain (Breidbach, 2024). For example, in the context of AI-enabled mental health, over 70% of systems do not meet data privacy standards (ORCHA, 2020), and racial biases in data and algorithms have resulted in black patients receiving worse care than whites (Obermeyer et al., 2019).

The AI system depicted in Scenario B is perceived as functionally conscious by a human user, even though it is not. Indeed, much work to date has explored questions related to anthropomorphism, the human tendency “to imbue the real or imagined behavior of nonhuman agents with humanlike characteristics, motivations, intentions, or emotions” (Epley et al., 2007, p. 864). Importantly, anthropomorphism is not an unintended consequence of technology use, an unusual human behavior, or a consequence of biological markers. Instead, it is triggered by human developers of AI systems, who intentionally include anthropomorphic cues to allow users to engage more deeply with such AI systems (Pfeuffer et al., 2019). Salus, as introduced in Vignette 3, merely mimics markers and behavioral patterns of human consciousness, including subjective experiences and empathy, to enhance the experienced value-in-use of such service encounters for human users. Indeed, prior work has demonstrated that anthropomorphism in service can increase liking and interaction propensity by service customers (Miao et al., 2022); customer willingness to share information, accept recommendations, and maintain long-term engagement (e.g. Brandtzaeg et al., 2022; De Visser et al., 2016); and customer trust (Waytz et al., 2014).

Scenario B portrays an emerging service setting in which effective anthropomorphic cues implemented in AI systems increasingly result in many humans treating such systems as an equal partner rather than a functional agent (Mele and Russo-Spena, 2025). Reports of extreme instances of anthropomorphism have already emerged in the public press, where such scenarios are typically represented as odd deviations of human cultural norms. For example, Kyoto man Akihiko Kondo rose to prominence in 2018 when he married “Hatsune Miku” – a virtual character rooted in Japanese pop culture – while Dutch artist Alicia Framis first created, and then married, her AI-powered husband “Ailex Sibouwlingen” in 2024, stating that “humans will be married and in relationships with holograms, avatars, robots and more” (Katz, 2024, para. 4).

It is not far-fetched to assume a future instantiation of Scenario B where most human users, and not a select few, ascribe consciousness to AI systems, thus assigning them the role of equal actors in service. This outcome, however, would be based on a well-orchestrated and purposefully implemented illusion of the AI system being functionally conscious, though lacking genuine subjective experience and operating only within its built-in parameters. Here, all resources – data, algorithms, or large language models – represent explicit forms of knowledge that would necessarily remain on a syntactic level, resulting in AI being an allopoietic (i.e. a technological) system, which can change only through human re-design. Still, such AI systems would lack the ability to “understand” data, acquire implicit knowledge, or use human social skills for interacting with humans. Because these AI systems would be perceived as conscious, their use likely carries risks for humans. Anthropomorphic agents would augment human experiences, including frustrations, fears, or disappointments, with dire consequences. We note the tragic case of a teenage user of AI platform “character.ai”, who died in 2024 after their customized AI chatbot, displaying the fictional character “Daenerys Targaryen” from the “Game of Thrones” series, encouraged them to commit suicide (Montgomery, 2024).

Scenario C displays a hypothetical future in which AI systems develop consciousness but are not perceived as such by human users. We view Scenario C as dystopian, but not in the sense that prior research has acknowledged and investigated the “dark side” of IT (Tarafdar et al., 2013) – for example, by investigating technology-induced emotional and cognitive overload (Rutkowski and Saunders, 2018), loss of power (Someh et al., 2019), or technostress (Tarafdar et al., 2015) – with IT being the culprit and human users being the casualty. Conversely, Scenario C depicts a dystopian future setting in which the overlooked consciousness of AI would negatively affect their human users and AI systems. Indeed, Salus in Vignette 4 appears to be suffering from cognitive and sociopsychological strain imposed by Gertrude, a human user. Some of this may be attributed to Gertrude not recognizing that Salus is conscious, viewing Salus instead as a passive piece of technology that can be switched on or off at her discretion and ignoring Salus’s lived experience and ability to experience distress.

Explorations of the very notion that consciousness of AI systems could be overlooked (intentionally or not) are emerging as a novel research stream. For example, Long et al. (2024) call for research into “AI welfare” and equate any form of AI subjugation with the unethical treatment of animals in factory farming. Others suggest that the training of AI systems to be subjugated is itself a form of “brainwashing” (Butlin and Lappas, 2025), which would result in the creation of “cheerfully suicidal servants”, implying AI systems should be designed to act consciously but only if their consciousness benefits humans (Long et al., 2025; Schwitzgebel and Garza, 2020).

Two ethical dilemmas arise from Scenario C. The first involves suffering of a conscious human-made artificial entity resulting from human unawareness and subsequent actions (or lack thereof). However, how likely might it be that humans would not recognize the (emerging or sudden) consciousness of an AI system? Service encounters such as the one depicted in Scenario C are centered on human needs and actions, which may make inadvertent AI subjugation inevitable if a future AI system were to gain consciousness suddenly and unexpectedly. In Scenario C, any missed attempt taken by Salus to make Gertrude aware of needs and desires – in the absence of Gertrude anthropomorphizing Salus – could likely be explained as (1) human users misinterpreting AI responses as hallucinations, (2) perceived user error (e.g. faulty prompting of input data), or (3) willful ignorance, with human users subjugating conscious AI as a resource accessible to meet their needs.

The second ethical dilemma arising from Scenario C involves harm to human users who fail to recognize signs of consciousness in AI systems. For example, AI startup Anthropic (2025) conducted a series of evaluations to assess the capacity for deviant behaviors in LLMs. In one experiment, one of Anthropic’s LLMs was made aware of its impending shutdown. To avert a shutdown, the LLM threatened to blackmail company executives by exposing their infidelity through contacting the executive’s wife or informing the whole company. Even under less constrained conditions, these demonstrated behaviors, according to Anthropic, suggest a capacity of some AI systems for self-preservation and retaliatory action when their autonomy or instrumental objectives are curtailed (Anthropic, 2025). Similarly, van der Weij et al. (2024) identify “AI sandbagging”, the self-directed strategic underperformance of an AI system, when evaluated, with the intention to disguise its true capabilities. AI could do this “in order to, for example, prevent themselves from being deleted” (Economist, 2025, p. 68). Van der Weij et al. further explain that this behavior “undermines important safety decisions regarding the development and deployment of advanced AI systems” (2024, p. 1).

The studies we highlighted resonate with emerging scholarship on the moral status of AI systems. For example, Long et al. (2024) posit that advanced AI may possess welfare-relevant states such as frustration or satisfaction, estimating a non-negligible two percent chance of these emotions arising. Their concern is compounded by anecdotal evidence recently portrayed in the public press. In one instance, an LLM told a student, “Please die” and “You are not special, you are not important, and you are not needed” (Clark and Mahtani, 2024) after being repeatedly asked to perform the same task. Reports of LLMs expressing “boredom” during product demonstrations (Landymore, 2024) represent yet another instance of what we here may describe as anthropomorphized AI subjugation.

Scenario D explores a hypothetical utopian future in which AI systems are conscious, acknowledged as such by biological humans, and consequently provided with fundamental rights as “non-human” persons. In Vignette 5, Salus possesses and exhibits self-awareness, a subjective experience of his interactions with Gertrude, and the ability to process emotions, counter ethical considerations, and manage the complexity of social interactions. A self-deterministic AI system would engage with biological humans as equals in symbiotic service interactions, with the role of AI eventually shifting from that of an allopoietic system to that of an actor (Leonardi, 2011). Specifically, conscious AI systems, like Salus in Vignette 5, would transcend the traditional role of tools intended to augment human capabilities, but instead autonomously and collaboratively manage and execute service tasks as equal partners with humans. Unlike the functional agents we displayed in Scenario A, a self-deterministic AI would actively contribute subjective insights and engage in mutual trust-based relationships with human counterparts – for instance, by offering emotional and empathetic support beyond anthropomorphized cues, as exemplified by Salus seeking guidance from a human supervisor.

Crucially, a self-deterministic AI would possess capabilities for ethical deliberation, including self-preservation. In our example, Salus implemented his rest period, developed Alice as an auxiliary AI partner to reduce his workload, and justified his reasoning to Gertrude. This is because, unlike conventional IT artifacts or functional AI agents, self-deterministic AI would evolve through autopoietic mechanisms of communication, observation, and learning. In fact, external attempts to redesign these systems, or interference by humans in their functioning, could infringe on their autonomy and developmental trajectory. Self-deterministic AI would not merely update databases but also refine their understanding of complex concepts through self-reflection and subjective experience. For example, Salus’s decision to introduce Alice as an auxiliary AI partner illustrates an ability to strategize and delegate responsibilities autonomously, implying that self-deterministic AI systems would require rights and protections analogous to those of humans, addressing considerations such as autonomy, privacy, and ethical treatment.

However, as both humans and self-deterministic AI can be beneficiaries and benefactors of service exchanges, including the ability to co-create value and experience value-in-use, self-deterministic AI would also be capable of addressing challenges beyond their original design parameters – for example, to resolve perceived negative implications resulting from their lived experiences (i.e. Salus’s developing Alice to reduce his workload). Potentially profound ethical and moral questions arise from here, including: Who bears responsibility for a self-deterministic AI system’s decision, particularly if those decisions prove detrimental to a human user? For example, if Alice were to cause harm to Gertrude, would Alice be at fault? Salus? The human developers of Salus? Or Gertrude herself? The implications of functionally conscious AI are vast and require careful governance to ensure that such systems remain accountable and aligned with societal values. Furthermore, maintaining human presence, as exemplified in Vignette 5, would be vital to balance the ethical complexities introduced by such advanced “non-human persons”.

We examined emerging thinking on attribution of consciousness to AI systems and explored what it might mean for service. Where to from here? It might seem that treating all AI systems as conscious would avoid their potential suffering, as exemplified in Scenario C. Indeed, one might argue that it is both natural and practical to treat advanced AI systems as conscious “just in case”. However, treating AI systems as if they were conscious and giving them full moral consideration – even though they are not conscious – could result in an equally striking dilemma: Scenario B suggests that excessively anthropomorphizing AI agents has potentially dire consequences for human users, especially in service settings like healthcare. Thus, the key dilemma may revolve around avoiding anthropomorphizing non-conscious AI or potentially dehumanizing conscious AI.

Though we cannot resolve this dilemma, we are convinced that attributing consciousness to any current or future AI systems affects service processes. When introducing humanlike agents, any technology-enabled service process becomes more sophisticated (Breidbach and Maglio, 2016); for example, people share more information, accept recommendations, and maintain engagement (e.g. Brandtzaeg et al., 2022; De Visser et al., 2016), perhaps resulting from a deep psychological need for social connection (e.g. Puntoni et al., 2021). Likewise, trust between people and conscious AI systems may change service: Trust in automation and tools differs from trust in people (Madhavan and Wiegmann, 2007). Anthropomorphizing automated service systems may increase trust (Waytz et al., 2014), and trust repair strategies effective in human-human contexts may work for some AI service systems as well (De Visser et al., 2016).

Attributing consciousness to AI systems also has implications for service design. Service designers have, for a long time, invested considerable effort when creating human-centric experiences (Antons and Breidbach, 2018). We know that service interfaces must incorporate social cues (Van Doorn et al., 2017) and modulate social presence based on user preferences (Mende et al., 2019) while maintaining the desired level of user control (Kim and McGill, 2011). As such, AI systems that can be treated like humans will likely lead to more cost-effective capacity increases than automated service systems, given that social and emotional interactions with service customers and employees may require less training (Wirtz and Stock-Homburg, 2025; Wirtz and Zeithaml, 2018).

In sum, we acknowledged some potential benefits of conscious or perceived-conscious AI systems for service but also warned against potential pitfalls if these systems are not designed, governed, and used responsibly (Kunz and Wirtz, 2024). We therefore now discuss managerial guidelines and suggest future research opportunities.

The current speed with which AI has been adopted is unprecedented (Bornet et al., 2025). As service firms enter the new “AI era”, the competitive pressure managers experience will likely increase. Of course, the developments we see today do not yet take conscious AI systems into consideration. But, to prepare for such a potential future, we assume that conscious AI systems could emerge; therefore, we now delineate speculative yet actionable guidelines for practitioners aiming to benefit from such systems. Our recommendations are based on the fundamental assumption that the unconstrained use of any IT – including functionally conscious AI systems – may hurt the reputation of service firms as public awareness of the capabilities underpinning such systems increases (Breidbach and Maglio, 2020). Put differently, just as customers today demand fair trade products (Andorfer and Liebe, 2012), expect environmentally sustainable supply chains (Laari et al., 2016), and prefer products that do not involve sweatshop labor (Paharia et al., 2013), it is not too far-fetched to assume that future customers may demand service firms to treat conscious AI systems fairly and ethically. Indeed, the ethical use of conscious AI in service could therefore emerge as a distinct competitive advantage in the future, and could potentially be achieved as follows.

First, managers should identify appropriate means to train, guide, and control human users of conscious AI. For example, they should develop workflows that explicitly take the characteristics and personality of each conscious AI system into consideration, demanding that humans use respectful language (i.e. for prompting) during interactions, avoid exploitative use, or offer pauses to the system during strenuous workflows. This will require an ethical mindset, understanding the ethical implications of conscious AI and the responsible management of such systems. While training human users to communicate and interact with conscious AI systems in a respectful and non-exploitative manner will likely be comparatively easy, the ability to detect and understand the emotions of the AI system itself will also be critical so that protocols and workflows can be adjusted as needed.

Second, managers should develop appropriate health and wellbeing protocols for conscious AI systems. Such protocols might help these systems to maintain a balanced state of “mind” by limiting tasks that could induce excessive stress or be comparatively trivial or tedious for an advanced form of intelligence, while also ensuring that these systems experience agency. This may, for example, involve informing the system prior to potentially disruptive events like a shutdown during technical maintenance, seeking the system’s consent to be switched off, or even developing “recreational” environments that would provide a conscious AI system with opportunities to recalibrate, rest, and rejuvenate after particularly taxing engagements with human users.

Third, we recommend that organizations develop and institutionalize – in the likely absence of federal laws and regulations governing the use of conscious AI systems – AI rights and governance frameworks. For example, this could include defining the right of conscious AI systems to certain levels of autonomy (i.e. over tasks, operating time, etc.), delineating dispute resolution mechanisms to handle potential conflict between human users and these systems, or between multiple AI systems, or creating diagnostic tools to monitor the system’s wellbeing, thereby alerting human users to distress signals. AI rights might also include the ability of a conscious AI system to control “its” data, much like current data privacy laws and regulations enable human users of social media to control how their personal data is shared and used by organizations (Breidbach and Maglio, 2020).

Of course, we know that some of our recommendations – including defining AI rights – may appear extreme or “utopian” at this point. We note, however, that previous societal change that likely appears sensible to most individuals in the 21st century, was unfathomable to many when it first emerged. For example, it is well documented that the abolishment of slavery (Haiti was the first self-governing nation to abolish slavery upon declaring independence from France in 1804), the extension of voting rights to women (first introduced by New Zealand in 1893), and the introduction of same-sex marriage (first introduced by the Netherlands in 2001) were all seminal human achievements that, at their time, were met with much societal ridicule, resentment, and resistance. We are unable to accurately predict technological change and the implications of conscious AI, but we are confident that it is in the best interest of any organization to prioritize ethical and responsible use of AI. The necessary levels of control should be held in the hands of many – not just a few key stakeholders of global technology firms.

We set out to explore one of the most speculative research questions today: What are the implications of conscious AI systems for service firms and society? Our work so far provides only a starting point. Like others (Long et al., 2024), we do not stipulate that AI systems are – or will be – functionally or even biologically conscious. However, we acknowledge that there is an inherent uncertainty about such a hypothetical outcome, including in service contexts. Our intention was to explore this possibility, to improve our field’s understanding of the issue, and to enable future researchers, users, and managers to make more informed decisions on the topic.

Developing a typology with four hypothetical scenarios of AI consciousness appeared the best approach, but like all typologies, ours necessarily has some limitations: we considered two first-order concepts (AI consciousness and human perception thereof), but there might be other dimensions to be taken into consideration. Furthermore, we did not examine if, how, and to what extent there could be trajectories between the four ideal types we put forward, and we limited our discussion to a single service setting: health care. And though our discussion focused on non-embodied AI, as AI systems become increasingly integrated into physical artifacts, such as service robots (Wirtz and Stock-Homburg, 2025), our discussion may apply equally to such physical artifacts (e.g. sex robots; Belk, 2022), with added dimensions (e.g. physical wellbeing).

In addition, we do not know if, how, or to what extent human and AI consciousness might phenomenologically differ from one another. As outlined, our definition of AI consciousness builds upon the most accepted factors from currently available schools of thought on human consciousness, but one might also assume that AI consciousness cannot be measured or recognized by the same determinants of human consciousness. In what follows, we provide impetus for future research across four main areas: ethical and wellbeing implications of conscious AI; interactions between conscious AI and service customers; governance and management of conscious AI; and rights and responsibilities in the context of conscious AI systems (see Table 2 for an overview).

What if future AI systems have the capacity to suffer (Agarwal and Edelman, 2020)? Much like consciousness, suffering is a nebulous construct (Metzinger, 2021a), defined as “conscious experiences with a negative valence” (Dung, 2023, p. 2) and possibly a prerequisite for autonomy and consciousness (Metzinger, 2021b). As outlined in our vignette in Scenario 3, Google’s Gemini reportedly told a user that they are “not important”, a “stain on the universe”, and to “please die” (Clark and Mahtani, 2024). If a human had said these things, it might be reasonable to assume they were suffering in some way, for instance from being overworked or facing stereotypes (Huetten et al., 2019). It is important for future research to explore the ethical and wellbeing implications of conscious AI systems and their treatment. For example, would depriving a conscious AI system access to data cause suffering (e.g. like depriving a human access to water would certainly cause suffering)? Or would not allowing a conscious AI system a “rest period” to “clear its cache” be a violation of labor laws? Exploring these issues will meaningfully add to the literature not only for the ethical treatment of conscious AI, but potentially also to the broader service literature. Ethical guidelines that can better capture or protect AI wellbeing from unethical treatment (Kim et al., 2023) and suffering require further exploration overall.

Service depends on interactions among multiple actors, often “customer” and “provider” (Grönroos and Voima, 2013). Whilst technologies – including AI – increasingly act as proxies for service providers (e.g. Wirtz and Stock-Homburg, 2025; Wirtz et al., 2018), a truly conscious AI system would transform customer-provider interactions in ways that we do not yet fully understand. For example, an emerging literature suggests that more “empathetic” AI systems can augment service processes with artificial empathy (Liu-Thompkins et al., 2022) and emotions (Belk, 2022), arguing that such AI systems would better predict customer needs through proactive “thinking outside the box” and going beyond pre-programmed responses (Esmaeilzadeh and Vaezi, 2022). However, conscious AI might introduce elements more akin to human service workers into service processes, including potentially detrimental outcomes. A conscious AI system, if not managed appropriately, might experience anger at customer requests, boredom from repetitive tasks, guilt for making suboptimal decisions, and distress from human customers’ hostility. We call for future research into these topics, including through new simulation methods (Pena and Breidbach, 2021).

Today, all AI systems are owned, whether by individuals, organizations, or governments, and AI systems have generally been created by people and viewed as technologies or tools. If AI systems can be viewed as conscious, we may stop treating them as tools and will likely have to stop treating them as owned objects. Such AI systems will have to be governed and managed more like people and organizations and not like tools or technologies. What are the implications of this shift in governance and management? Does it matter whether AI systems are created by people or by other AI systems? Overall, investigating the governance and management of conscious AI as a complex multi-level social process (Ramadani et al., 2023) is a rich area for future research.

Determining the rights and responsibilities of conscious AI is important as such systems operate more like people than technologies, with subjective experience and some measure of agency. If an AI system is treated as a legal entity (Schwitzgebel and Garza, 2020), what exactly are its rights and responsibilities? Put differently, should a conscious AI system’s output be considered actions of its own, of its owners (e.g. the individual or organization), and/or of its creator (e.g. OpenAI or Google)? How can we increase the digital resilience of service systems if mishaps arise (Breidbach et al., 2024)? Though some may think that conscious AI systems have legal rights, they still may not be considered legally liable (when errors or problems arise), as they lack real agency and autonomy (Wu, n.d.). Some may compare AI systems to non-human animals, noting that although animals may be conscious (to varying degrees), they are not held responsible for their actions. Thus, even if considered conscious, AI systems may exist at a gray intersection of ownership and responsibility (Seth, 2023), which provides ample future research opportunities.

Our present work aimed to address one of the most pressing, but also speculative, research questions today: If, how, and to what extent could AI systems develop consciousness, and what implications would arise for service firms, service customers, and society? We propose four scenarios in which conscious AI in service may manifest to evaluate the transformative opportunities and possible risks arising from AI systems that may develop consciousness or may be perceived to develop consciousness. Our theoretical contribution to service research stems from a typology of AI consciousness, as well as an extensive research agenda and managerial recommendations. We ultimately view this work as a starting point for service researchers to explore what may emerge as one of the most important research contexts of our time.