The effects of chatbots’ attributes on customer relationships with brands: PLS-SEM and importance–performance map analysis Open Access

Technological advancements are changing the ways through which firms can manage their customer interactions and, consequently, the customer experience (Chung et al., 2020). Among the emerging technologies, Artificial Intelligence (AI) is considered a particularly disruptive technology capable of radically changing firm–customer relationships in every sector (Campbell et al., 2020). Kaplan and Haenlein (2019, p. 15) define AI as a “system's ability to correctly interpret external data, to learn from such data, and to use those learnings to achieve specific goals and tasks through flexible adaptation systems.” The underlying idea of AI is that, thanks to the use of software and hardware, firms can analyze data and provide real-time interactions with customers, making technology-based interactions more human and customer-centric (Hoyer et al., 2020; Libai et al., 2020). Therefore, many marketing opportunities can derive from AI applications (Martínez-López and Casillas, 2013).

Consequently, many firms are investing in digital services to improve customers’ experiences. Virtual service agents, or “e-service agents,” such as chatbots (an AI application), are examples of these efforts (Trivedi, 2019). A chatbot is an instant-chat service able to operate similarly to an offline service agent (Chung et al., 2020). The most popular chatbots include personal assistants like Alexa, Siri and Cortana. Indeed, a chatbot interacts in a familiar way with consumers, and its responses can consist in voice or text messages, images and so on. Like those of offline service agents, the roles of chatbots are becoming central in determining customer satisfaction. Indeed, chatbots represent the brand in customer relationships (Chung et al., 2020; Zarouali et al., 2018).

However, despite the increasing relevance of this topic, academic research into the role of chatbots in influencing customer satisfaction still remains scant (Hoyer et al., 2020). Specifically, there is a need to assess how chatbots should be designed to satisfy customers and improve their attitudes toward the brands represented. This study addresses this gap and investigates whether perceived hedonic and utilitarian attributes of chatbots can influence customer satisfaction and, consequently, the customer–brand relationship. For this purpose, a model comprising both the hedonic and utilitarian attributes of chatbots is developed and estimated. Moreover, to gain more detailed understanding and enrich the practical implications of the findings, the estimation of the model is complemented with importance–performance map analysis (IPMA).

The remainder of the paper is structured as follows. First, a review of previous studies on how chatbots affect customer satisfaction is provided. Next, the research model and hypotheses are presented, followed by a description of the methods and results. A discussion of the findings and conclusions completes the paper.

Technology has assumed a key role in enabling companies to achieve competitive advantage (Chaiprasit and Swierczek, 2011). At the same time, in today’s highly globalized competitive environment, companies are paying extreme attention to total quality management (TQM) methods, tools and techniques. Continuous improvement, innovation and standardization through TQM play a key role in increasing competitiveness (Tasleem et al., 2019).

Following this perspective, from the management point of view, it therefore becomes essential to integrate elements of technology management (TM) and TQM. The first is related to dimensions such as product technology, process technology and information technology and more recently industry 4.0 and Industrial Internet of things. The second is both a philosophy and a strategy oriented toward continuous change (de Souza et al., 2021).

Although there is still no agreement on the factors that constitute TQM, many researchers have attempted to overcome this lack of consensus through the use of a multidisciplinary approach. This work has led to the identification of the following six factors as the elements that make up TQM: leadership, strategic planning, customer focus, workforce focus, process management, and information and knowledge management (Agarwal, 2017). Hence, again, technology plays a fundamental role for quality management and improvement. Moreover, these factors represent the components of one of the world’s major awards for excellence namely the Malcolm Baldrige National Quality Award, established by the U.S. Congress in 1987. The award aims to promote a culture of quality, raise awareness of the importance of quality management and give recognition to organizations that have implemented a successful quality management system (Tasleem et al., 2019). TQM and technology are thus becoming two key elements of successful organizations that complement and influence each other (Brah and Lim, 2006; Chiarini, 2020; Prajogo and Sohal, 2006).

Among the multiple technologies which have the potential to contribute to quality improvements, chatbots are exponentially gaining popularity in many sectors, such as education and health (Laranjo et al., 2018; Pérez et al., 2020). However, it is in the area of firms’ customer services that chatbots have the greatest application. In this context, as the competition has increased, providing quality customer service has become a strategic element for a firm’s success (Scheidt and Chung, 2019). Thus, service agents who personally interact with customers in representing the brand are central in solving customer problems and play fundamental roles in determining customer satisfaction. Due to the advent of digital technologies, firms increasingly are shifting to digital services, and the roles of service agents are changing profoundly. Indeed, many firms are transforming their traditional customer-service approaches to digital methods (Cheng and Jiang, 2022). E-service agents, such as chatbots, are new technology tools that attempt to satisfy customers in a similar way as offline service agents (Chung et al., 2020). Indeed, chatbots are virtual assistants that simulate human conversations, not only by providing information but also by interacting, using a familiar language and attempting to transmit emotions (Hoyer et al., 2020; Schmitt, 2019). Clients can interact with e-service agents from any location on a 24-h basis (Cheng and Jiang, 2022). As a result of this digital revolution, people must increase their technological ability, while the technology itself must become humanized (Schmitt, 2019). Chatbots are crucial in determining customer satisfaction and, therefore, in enhancing the brand relationship.

To analyze these effects, we follow the Consumer Acceptance of Technology (CAT) model (Kulviwat et al., 2007). Unlike the traditional Technology Acceptance Model (Davis, 1989), which considers only cognitive elements, the CAT model also includes affective elements. This aligns with the work of several researchers (Fiore et al., 2005; Nasco et al., 2008), who suggest that in consumer relationships, technology must reach two types of goals: utilitarian and hedonic. Utilitarian goals are guided by cognitive elements and oriented to problem-solving (Dhar and Wertenbroch, 2000). These components are connected strictly to the technology’s analytical characteristics. They represent the value derived from elaborating on the information received by the chatbot (Hoyer et al., 2020). Hedonic goals are related to affective aesthetics – fun and enjoyable elements (Batra and Ahtola, 1991). They represent the value that consumers receive from emotional stimulation (Hoyer et al., 2020). In our model, we identified two utilitarian elements in particular – information quality and system quality – and one hedonic element related to the chatbot experience.

Information quality represents the semantic success of the technology (DeLone and McLean, 1992). In general, the concept encompasses both the intrinsic and extrinsic elements of information quality. Specifically, the term “intrinsic elements” refers to objective elements, such as the provision of correct, credible and congruent information. These are important aspects that cannot, however, be separated from extrinsic considerations relating to the context in which they are applied (Lee et al., 2002) and representational aspects (Wang and Strong, 1996). Contextual elements involve the completeness and currency of information. From this point of view, one must evaluate the quality of the information to the users and whether the information provided by the technology is capable of helping them complete an activity – for example, making a decision (Nelson et al., 2005). The representational component involves the way in which the presentation of the information (the format) allows the receiver to better understand and interpret the information itself. In conclusion, information quality encompasses the accuracy, currency, completeness and format of the information, shaping perceptions of quality in the context of use. Following this reasoning, the information provided by a chatbot should be relevant, correct, accurate, credible, and, of course, useful (Chung et al., 2020; Zarouali et al., 2018). The literature also highlighted that poor information quality can diminish the total performance of a firm by increasing costs (Swanson, 1997). Thus, the role of the communicator – even if it is computer-mediated – becomes fundamental. In any case, consumers must have the perception that the chatbot is able exactly to understand their problems and provide appropriate answers. Chatbots must therefore be credible, experienced and competent (Chung et al., 2020). According to Trivedi (2019), the quality of information offered by chatbots is critical in determining the customer experience. If the information is not correct, accurate or up-to-date, it can lead to a negative perception of the entire business and in particular to the belief that the company’s offerings (of products/services) are of limited quality (Gao et al., 2015). Based on these arguments, we hypothesize the following:

System quality is related to the technical aspects of a chatbot. In particular, the quality of a chatbot is determined by aspects such as usability, reliability, availability, adaptability and timeliness (Trivedi, 2019). “Usability” refers to the ease of use of a chatbot. In particular, if consumers perceive a chatbot as difficult to use, it can negatively influence customer satisfaction. Reliability involves the ability to interact with the chatbot continuously, at any time and in any place. “Adaptability” refers to the capacity to keep up with changing developments. Simultaneously, consumers expect that a chatbot’s answer is given in a couple of seconds. If the time is too long, this can negatively influence customer satisfaction (Chung et al., 2020; Trivedi, 2019; Zarouali et al., 2018). Therefore, we hypothesize the following:

In general, customer experience has become a key factor for the success of a business. In the digital age, when enterprises must be continuously available 24 h a day, optimizing and improving online experiences are critical today more than in the past (Lemon and Verhoef, 2016). Indeed, consistent with offline occurrences, Rose et al. (2012) highlight how important emotions are, even in the online context. In this context, online communication becomes essential in enhancing the customer experience. Experiences with chatbots are related to the hedonic goal of using a technology – that is, to be engaged in an emotional experience. An emotional experience involves the enjoyable aspects of interaction: pleasure, arousal and dominance. “Pleasure” refers to the pleasantness or enjoyment of a chatbot conversation, but emotional experience involves not only enjoyment but also the arousal of being involved in a mentally stimulating conversation (Zarouali et al., 2018). “Domain” (“dominance”) refers to the fact that consumers, when interacting with a chatbot, feel that they can act freely and are in absolute control of their actions (Coyle et al., 2012). Previous studies have highlighted that these aspects can determine whether consumers will respond positively to e-service agents (Godey et al., 2016). Therefore, we hypothesize the following:

Finally, we know that when a product or a service meets customers’ expectations, the customers are satisfied (Wiedmann et al., 2009). Therefore, customer satisfaction derived from interactions with a chatbot can enhance and empower the quality of the overall brand relationship. Hence, we suggest the following:

In sum, as shown in Figure 1, following the CAT model in our research, we evaluate the effects of cognitive elements on customer satisfaction (H1 and H2) and the effects of emotional elements on customer satisfaction (H3). Finally, we test the effects of customer satisfaction on quality (H4).

To achieve our research goals, we conducted a questionnaire-based survey among a sample of Italian consumers. Data collection took place in April 2021. The questionnaire was distributed online through the personal networks of the authors, relying on a convenience sampling technique. Overall, we received 275 questionnaires; however, 19 were excluded from the analysis because respondents had no experience with chatbots. Hence, the final sample was composed of 256 participants. Table 1 summarizes the main characteristics of the sample. Participants reported using chatbots mostly to ask information or ask for assistance. Chatbots that were mostly used by participants belonged to telecommunications and technology, while chatbots from industries such as personal (health)care and fashion were rarely mentioned. While these data reflect the respondents’ actual experience, they also suggest that the results of our analysis cannot be directly generalized to all industries.

The questionnaire included multiple-item measures for each construct developed from previous studies. Specifically, information quality, system quality and customer experience with the chatbot were measured, using four, five and three items, respectively, from the study by Trivedi (2019). Customer satisfaction was determined on the basis of four items from Chung et al. (2020). Finally, three items from Algesheimer et al. (2005) were used to measure brand relationship quality. Respondents were asked to refer to their latest experience with a chatbot and then give their ratings. All items were measured on five-point Likert scales, with extremes being 1 = totally disagree and 5 = totally agree. Consistently with the original scales, constructs were modeled as reflective. Table 2 shows the complete list of items.

Data were analyzed using Partial Least Squares Structural Equation Modeling (PLS-SEM) (Hair et al., 2020). The analysis was conducted using the software SmartPLS 3 (Ringle et al., 2015). PLS-SEM is a nonparametric method and, unlike covariance-based structural equation modeling, makes no distributional assumptions (Hair et al., 2017). This method is suitable when the purpose is the assessment of a model’s predictive power, the main focus of this study (Hair et al., 2019).

The “standard” PLS-SEM estimations was then complemented with an IPMA. This is a well-established method of analysis to assess customer acceptance of specific features of the offering and is based on the assumption that “consumer satisfaction is a function of both expectations related to certain important attributes and judgments of attribute performance” (Martilla and James, 1977, p. 77). Available studies particularly emphasize how IPMA can provide managerially relevant insights, helping organizations to prioritize important attributes to work on (Phadermrod et al., 2019). Specifically IPMA has been extensively applied to quality improvement (Roy et al., 2020) and to technology design, such as in the case of mobile applications’ features (Chen et al., 2016). Prior applications of IMPA have shown that direct measures of attributes’ importance – i.e. measures obtained from participants’ ratings using Likert scales – can provide biased findings (Oh, 2001). Thus, indirect measures of importance such as those derived for example from correlation or multiple regression analysis have been recommended (Azzopardi and Nash, 2013). Therefore, in this work, we used an indirect measure of importance, which is equal to the total effects on the target construct, provided by the PLS-SEM analysis.

IPMA has been recently applied in the context of PLS-SEM to enable researchers to gain richer and more precise insights from their findings because it simultaneously considers both the path coefficients estimates and the average values of the latent variable scores (Ringle and Sarstedt, 2016). In our study, the “standard” PLS-SEM analysis allowed to understand the magnitude of the effects of the three independent variables information quality, system quality and experience with chatbot on customer satisfaction. However, this analysis did not evaluate the average values of these three independent variables. In other words, it did not consider whether, according to participants’ ratings, chatbots performed well or not in terms of information quality, system quality and experience. The joint evaluation of these constructs’ importance (i.e. of their effects of customer satisfaction) and performance (i.e. of their average values) enabled us to completement “standard” PLS-SEM results with relevant insights to guide managerial action (Hair et al., 2018).

The final output of the IPMA is a map in which the x-axis shows the importance and the y-axis the performance of each attribute. In particular, the performance latent results from the rescaling of each attribute’s average scores on a scale from 0 to 100 (where 0 and 100 indicate the lowest and highest levels of performance, respectively). The map can further be divided into four areas through the addition of a vertical line representing the mean importance value and a horizontal line depicting the mean performance value. As a result, each attribute will be placed within one specific area characterized by a certain level of importance (low or high) and a certain level of performance (low or high) (Hair et al., 2018).

Finally, we extended IPMA on the indicator level, meaning that we examined the importance and performance of each of the items used to measure the three independent variables (Table 2). This additional level of analysis made it possible to identify more specific areas of intervention (Hair et al., 2022; Ringle and Sarstedt, 2016).

All constructs’ measurement models were specified as reflective. Hence, they were evaluated based on outer loadings, internal consistency reliability, convergent validity and discriminant validity (Hair et al., 2020). As shown in Table 2, outer loadings were above the recommended value of 0.707, with only one exception, which was nonetheless very close to that value (BRQ1, 0.694). Hence, the underlying factor explains more than 50% of each indictor’s variance. Next, the internal-consistency reliability was assessed (Table 3). For all latent variables, the values of Cronbach’s alpha, exact reliability, ρ_A and composite reliability were greater than 0.70, showing that the internal-consistency reliability was met (Dijkstra and Henseler, 2015; Hair et al., 2017). Moreover, convergent validity was assessed because, for all constructs, the values of the average variance extracted (AVE) were greater than 0.50 (Hair et al., 2019). Finally, discriminant validity was met as well because the square root of each construct’s AVE was greater than its highest correlation with any other construct, as requested by the Fornell–Larcker criterion (Fornell and Larcker, 1981) (Table 4).

After we successfully assessed the measurement models of the five constructs, we evaluated the structural model. First, we checked the absence of collinearity issues by inspecting the values of the inner variance inflation factor. All the Variance Inflation Factor (VIF) values are well below 5 (the highest being 2.28), highlighting that collinearity was not an issue (Hair et al., 2019). We then assessed the relevance and significance of the structural-model relationships based on the bootstrapping routine (5,000 subsamples, bias-corrected and accelerated bootstrap, two-tailed test). Table 5 and Figure 2 provide the detailed results of the estimations.

The analysis showed that information quality (β = 0.721, p < 0.01) and experience with the chatbot (β = 0.239, p < 0.01) had positive effects on customer satisfaction with the chatbot. Therefore, both H1 and H3 were supported. However, system quality had no significant effect on customer satisfaction with the chatbot (β = 0.046, p > 0.10) and was thus rejected. Finally, the findings supported H4, indicating a positive effect of customer satisfaction on brand-relationship quality (β = 0.748, p < 0.01).

The results of the IPMA allowed to enrich the understanding and interpretation of these findings (which highlighted the importance of each construct) by considering also the performance of the constructs. As shown in Figure 3, system quality is the attribute registering the lowest importance and the higher performance. Hence, investing on system quality improvements should not be the top priority for chatbots’ designers. The high level of performance reported by system quality may to a certain extent explain its nonsignificant effect on customer satisfaction. In fact, good system quality may have been taken for granted by chatbots’ users, thus causing a sort of ceiling effects (i.e. when the independent variable is above certain levels, it has no more effects on the dependent variable). On the contrary, the performance of the two most important attributes (information quality and experience with chatbot) is below the average value of performance, indicating that there is room for improvement. Specifically, a one-unit increase in information quality’s performance (from the current level of 61.65–62.65) would increase customer satisfaction by 0.72 points (from the current level of 58.94–59.66). A one-unit increase in customer experience’s performance would increase customer satisfaction by 0.23 points. Therefore, chatbot designers should prioritize actions to increase information quality.

Finally, Table 6 shows the results of the IPMA on the indicator level. These data highlight the specific aspects to improve. Among the items that measure information quality, IQ3 shows a high level of importance and a relatively low level of performance, suggesting that designers could focus on the improvement of the quantity of information provided by the chatbot. On the contrary all system quality’s items have high levels of performance and low levels of importance. These findings indicate that currently there is no urgency to enhance these aspects.

To complete the PLS-SEM analysis we evaluated the model explanatory power. The coefficients of determination R² of the endogenous constructs were high. Specifically, for customer satisfaction, R² was 0.854, and for brand-relationship quality, R² was 0.559. Hence, we concluded that the model had high explanatory power. Finally, we assessed the model’s predictive power using the PLSpredict routine instead of the blindfolding procedure, as suggest by recent methodological works (Sarstedt et al., 2022a, b). We ran PLSpredict with 10 folds and 10 repetitions. The findings showed that, for all items except one (BRQ1), the PLS-SEM estimation generated a lower prediction error (root mean squared error) compared with the linear-model benchmark. In addition, all the items measuring brand relationship quality had a value of Q²_predict higher than 0. This analysis confirmed that the model had suitable predictive power (Shmueli et al., 2019).

The results of this study enhance the available knowledge concerning the effects of e-service agents (chatbots) on customer satisfaction and on customer–brand relationships. In particular, while previous studies have approached the topic from the perspective of technology acceptance (Meyer-Waarden et al., 2020; Murtarelli et al., 2022), the present study represents one of the first concrete attempts to assess the impact of chatbots on a brand relationship. Therefore, the results of the study allow us to obtain significant information related to the effects after actual use of this technology in customer services.

Regarding the utilitarian (cognitive) elements, our study confirms the importance of the information quality provided by chatbots. At the same time, unlike other studies (Trivedi, 2019), in our work, the technical element is not important in determining customer satisfaction. However, as noted in our analysis, this unexpected finding may be related to a sort of ceiling effect. Regarding the hedonic (affective) elements, our study confirms the role of the emotional experience in determining customer satisfaction. Information quality and emotional experiences with chatbots are crucial in determining customer satisfaction and, finally, enhancing the brand relationship. Therefore, while e-service agents are typically the results of technological advancements, firms must not forget what consumers truly require from service agents: the quality of information and an emotional experience. Consumers do not expect technical perfection, but, overall, consumers appear interested in the quality of the information received and in the emotions derived by their relationships with chatbots. The results confirm the trend to humanize the technology. Therefore, firms are encouraged to consider both the utilitarian and hedonic aspects of the consumer experience carefully when designing their chatbots. In addition, from the managerial point of view, the results also confirm the need to integrate elements of TQM with elements related to TM. Thus, it becomes essential in the perspective of continuous improvement of customer services to take into account the growing importance of the impacts of these new technologies.

Of course, this study presents several limitations. More data should be collected to corroborate the results. In the future, it will be useful to deepen the analysis by comparing the estimations in different sectors to identify whether the roles of chatbots change in relation to the sector (e.g., the advanced technological versus the traditional sectors). Customer perceived importance of chatbots’ attributes may also vary depending on the industry. For example, system quality may register higher importance in the health-care industry (May and Denecke, 2022). It also will be useful also to repeat the survey in the future in different countries to capture potential differences.