Leveraging AI for accurate supply chain risk classification: optimizing the operational parameter space of LLMs

Efficient and effective Supply Chain Management (SCM) means the assurance of optimal flow of products and materials throughout a business’s value chain while maintaining a holistic profitability and quality. This quest however, has become increasingly challenging in recent years due to an unprecedented range of risks and disruptions impacting supply chains on a broad scale. While supply chains have stood and arguably always stand under a general level of pressure because of common market volatility and uncertainty, studies show that the list of disruption-inducing drivers and risks has been growing significantly (KMPG & ASCM, 2024). These new drivers and risks range from structural rather long-term and often foreseeable but hard-to-solve aspects such as labor scarcity to ad hoc events such as the Covid-19 pandemic or the war between Russia and Ukraine which were unexpected for many. Due to the nowadays globalized nature of many supply chains, the impacts of such events were and still are felt on a world-wide scale (Gurtu and Johny, 2021). For example, the Covid-19 Pandemic largely contributed to health risks, workforce deficiencies, and substantial shifts in demand in many markets (Handfield et al., 2020). But also smaller scale disruptions such as man-made events like the Ever Given Grounding in 2021 or the damaged railway tunnel in Rastatt, Germany in 2017 showed the strong influences within bottlenecks in transportation systems pushing companies into a more reactive rather than proactive state.

As a result, striving for increased supply chain resilience, the discipline of Supply Chain Risk Management (SCRM) has moved to the centre of interest and attention throughout the research and practical landscape (Deiva Ganesh and Kalpana, 2022a). SCRM encompasses the systematic and phased identification, evaluation, mitigation, and monitoring of risks that could potentially affect a business’s supply chain (Aqlan and Lam, 2016). Here, the accuracy and timeliness of risk identification, representing the foundation and first step within the process, plays a pivotal role (Deiva Ganesh and Kalpana, 2022b).

In this context, the recent rise of Artificial Intelligence (AI) has led to new ways and opportunities to cope with huge amounts of structured and unstructured data potentially indicating supply chain risks (Deiva Ganesh and Kalpana, 2022a). Initial research exploring the integration of AI into SCRM and thus going beyond conventional approaches highlights first promising use cases such as the prediction of uncertain events (Schroeder and Lodemann, 2021), continuous monitoring (Chu et al., 2020), or extraction of unstructured data (Modgil et al., 2022). Yet, due to the novelty and ongoing advances of AI technologies and their capabilities, this field of research is still considered underrepresented and is lacking formal approaches (Richey et al., 2023). Furthermore, in the realm of AI-based risk classification utilizing Large Language Models (LLMs), prevailing research predominantly concentrates on classifying risks within overarching categories, such as environmental risks and financial risks (Chu et al., 2020). The utilization of LLMs for risk classification however, holds the potential to go far beyond broad, rather static classification by dynamically classifying and thus identifying individual supply chain risks in real time based on extensive sources, such as social media or newspaper articles. This approach has the capacity to augment existing supply chain risk identification practices by offering a more granular and timely assessment. Here, due to the sensitivity of the task and the necessity of high accuracy, optimization of the LLM set up by means of operational parameters such as Temperature and Top P, is imperative (Xu et al., 2022).

In this regard, this paper aims to further the current field of research of AI-driven supply chain risk management, or more particularly, risk identification by introducing an LLM-based supply chain risk identification framework named SCREWS (Supply Chains Risk Early Warning System) and exploring and analyzing the LLM operational parameter space for the use case of accurate risk classification.

To reach the given objective, the remainder of this article is structured as follows: Section 2 overviews the current research landscape by means of a comprehensive systematic literature review (SLR) aiding to further sharpen the research agenda and to derive the research questions. In Section 3, the followed research methodology, a Design of Experiments Approach (Doe) as defined by Kleppmann (2020) is presented. The Doe is subsequently operationalized throughout Section 4 including the initial introduction of the risk identification framework SCREWS and the outline of the optimal set up for the use case of supply chain risk classification within the showcased framework using up-to-date sample data from different digital news outlets. Section 5 then highlights the practical implications after which limitations are discussed and a conclusion is given in Section 6. This study focuses on the operational parameter space of LLMs for binary supply-chain risk classification under a fixed model architecture.

To systematically and transparently assess the current state of the field, the SLR framework of vom Brocke et al. (2009) is applied. Ensuring rigor in documentation and target-orientation, the SLR framework employs five distinct phases: definition of review scope (1), conceptualization of topic (2), literature search (3), literature analysis and synthesis (4), and research agenda (5) (see Figure 1).

The following sections address the conduction of the phases against the background of the present research.

To precisely delineate the review scope and thus ensure coherence and purpose-driven execution, the established taxonomy of literature reviews by Cooper (1988) consisting of six characteristics is leveraged as suggested by vom Brocke et al. (2009).

As summarized in Figure 2, the present SLR focus (1) is versatile: It concentrates on research outcomes as well as the methods applied and their application allowing to generate an overview of the extent of already conducted LLM parameter optimization research and its application areas. The main goal (2) is to consolidate and integrate the findings. Furthermore, the SLR is organized (3) on a conceptual level (that is grouping of similar ideas), while a neutral position (4) is taken. As this review aims to assess the current field of academic research regarding the given topic (cf. Section 1), scholars are defined as the main audience (5). The present article is rooted in the domain of Supply Chain Management. Therefore, the audience is further narrowed down to scholars specializing in this field. Additionally, the increased transparency over the LLM applicability in the context of supply chain risk management and classification is also considered relevant for practitioners, namely supply chain managers, particularly risk managers, as it holds the potential to serve as basis for the practical implementation. Lastly, in terms of coverage (6), an exhaustive coverage with selective citation is followed.

Serving as basis for both a unified understanding and groundwork for the subsequent literature search, terms central to the given topic, namely Supply Chain Risks, Supply Chain Risk Management, Artificial Intelligence, and Large Language Models are briefly outlined in the following.

While the term risk is commonly used and easily understood in everyday language, the scientific concept of risk has been subject to various definitions, interpretations, and change also depending on the context (Heckmann et al., 2015). Ivanov (2021) describes risk as “a measure of the set of possible (negative) outcomes from a single rational decision and their probabilistic values”. In simpler words, risk can be summarized as the product of an event’s probability of occurrence and the corresponding consequences (Deiva Ganesh and Kalpana, 2022a). In contrast to this, the term uncertainty, which is often used in relation to risk, refers to a broader perspective, where also events causing positive deviations from an expected outcome are included (Ivanov, 2021).

In the domain of supply chains, risks can be defined as the combination of an event’s likelihood of occurrence and its consequences on any part of a supply chain resulting in operational, tactical, or strategical disruptions or irregularities (Ho et al., 2015).

Similar to the definition of the term itself, the classification of supply chain risks does not follow a unified understanding, either. For example, Truong Quang and Hara (2018) distinguish seven different risk categories (external risks, time risks, information risks, financial risks, supply risks, operational risks, and demand risks), while Tang (2006) narrows it down to two categories (operational risks and disruption risks).

Combining the research of Chopra and Sodhi (2004), Tang (2006), Tang and Nurmaya Musa (2011), Ho et al. (2015), and Truong Quang and Hara (2018) this study follows the supply chain risk model as proposed by Ivanov (2021) depicted in Figure 3.

Supply Chain Risk Management (SCRM) signifies a subfield of Supply Chain Management (SCM). As already mentioned in Section 1, it represents the approach of the systematic and phased identification, evaluation, mitigation, and monitoring of the above-highlighted risks that could potentially affect a business’s supply chain (Aqlan and Lam, 2016). Due to so-called cascading effects or ripple effects, which can occur up- and/or downstream the supply chain or within a whole supply network, the initial risk identification portrays a highly challenging endeavor (Cigolini and Rossi, 2010; Ivanov, 2021). Overall, effective SCRM aims to reduce vulnerabilities and improve resilience and robustness of the supply chain (Deiva Ganesh and Kalpana, 2022a).

Due to the recent AI hype within research and practise and the resulting rapid technological advancements, the lines between what can be considered an AI and what cannot are blurred. Overall, Artificial Intelligence (AI) can be described as an umbrella term for technologies and techniques that enable machines to fulfill complex tasks typically associated with intelligent beings (Sheikh et al., 2023). A particularly relevant topic in this domain is Machine Learning (ML). ML, being the underlying technique of many modern AI applications, enables computers to learn patterns and make decisions or predictions based on structured and unstructured data without being explicitly programmed (Rebala et al., 2019). It uses algorithms to self-improve its performance over time by adapting to new data, mimicking how humans learn from experience (Rebala et al., 2019).

Leveraging Natural Language Processing (NLP) and Generation (NLG), Large Language Models (LLMs) are characterized by their ability to perform a wide range of tasks (e.g. summarization, translation, classification, or sentiment analysis) using qualitative often unstructured data, namely text. From a technical perspective, a so-called transformer serves as backbone for many modern, high performing LLMs such as ChatGPT or Gemini. As introduced by Vaswani et al. (2017), transformers are deep-learning architectures designed to handle sequential data efficiently by using a self-attention mechanism. The output quality of an LLM is highly dependent on the underlying training data, the given prompt as well as the applied operational parameters (e.g. Temperature, Top P, Max Tokens) against the background of a given use case. Especially the detailed finetuning of the model’s parameters is considered crucial as already outlined in Section 1.

Phase 3 encompasses data base selection and the according keyword, backward, and forward search as well as a parallel evaluation of sources (vom Brocke et al., 2009). The keyword combinations used aim to uncover publications related to Supply Chain Risk Classification including the methods and approaches addressed (cf. Figure 4). Based on this, the extent to which LLM-based classification and optimization is covered can be assessed and discussed serving as groundwork for the remainder of this paper.

For increased currency, relevance, authority, accuracy, and purpose amongst the identified publication items, the following inclusion and exclusion criteria are applied:

Inclusion criteria:

1. Keywords 1 and 2 are present in title (cf. Figure 4)

2. Articles published in scholarly journals or proceedings of renowned conferences, to increase the likelihood of high quality due to peer-review processes

3. Publication after 2019

Exclusion criteria:

1. Bachelor and master theses, patents, citations

2. Full texts not available

3. Publication not written in English language

4. Duplications

As depicted in Figure 5, the according search procedure within 4 distinct academic databases resulted in 2.905 identified articles of which 38 were determined relevant based on initial analysis.

To assess the current research landscape, a concept matrix, as proposed by vom Brocke et al. (2009), is developed based on the analysis of the 38 selected articles (see  Appendix). The matrix reveals that existing studies encompass a broad range of topics within the domain of supply chain risk identification. Overall, in terms of the research scope, the literature can be divided into two different categories: (1) a static, generalized, and often one-time identification of supply chain risks and (2) a dynamic, targeted identification of risks, frequently at the level of individual focal firms, which represents the primary focus of this paper.

Static risk identification is predominantly conducted leveraging systematic literature reviews sometimes complemented by expert interviews and surveys (e.g. Rosales et al., 2019; Kusrini et al., 2021; Ramiah et al., 2022). Additionally, on the basis of this approach, deep dives into particular focus areas such as certain industries, supply chain operations, or risk types are given: For example, Zhao et al. (2024a) and Rosales et al. (2019) examine risks associated with the agri food industry while further studies explore the solar industry (Ramiah et al., 2022), or the energy industry (Zhang and He, 2024). In the context of supply chain operation-specific analyses, Mismar et al. (2022) highlight risks within last mile-processes, while Panjehfouladgaran and Lim (2020) outline risks related to reverse logistics activities. Furthermore, Pandey et al. (2020) outline particular insights into risks grounded in cyber security.

With regards to the dynamic, more individual risk identification studies, aiming for an increased degree of information coverage, accuracy and proactivity, a wide range of advanced methods and models have been employed, often introducing innovative tools and approaches. For instance, Liu et al. (2024) utilize Big Data Analytics (BDA) and Machine Learning (ML) in a two-step procedure that integrates a generative adversarial network, a stacked auto-encoder, and a deep neural network to achieve precise risk classification. Similarly, predictive approaches are emphasized by Rezki and Mansouri (2024) and Nagy et al. (2022), enabling the identification of potential risks before they materialize. Notably, Aboutorab et al. (2022) introduce a “Reinforcement Learning approach for Proactive Risk Identification” (RL-PRI), showcasing the potential of adaptive AI-driven methods. Additional frameworks include the Fuzzy Inference Decision Support System (FIDDS) proposed by Salamai et al. (2019), a dynamic voting classifier developed by Salamai et al. (2021), and a hybrid concept integrating network theory with a cascade failure model, as outlined by Wang and Zhou (2024).

Moreover, due to the pressing need for handling unstructured data as already mentioned in Section 1, a few studies explore opportunities for LLM-integrating solutions. Shahsavari et al. (2024) present a solution by means of their CERIA framework (Contributing Event-based Risk Identification and Assessment), which further integrates their LUEI framework (Lightweight Unsupervised Event Identification) within its modular setup. Here, LLM capabilities are leveraged for analyzing news article content, extracting relevant event data such as location and date, and outlining cause-and-effect relationships. Next to this, Shishehgarkhaneh et al. (2024) introduce LLM-based recognition of entities related to supply chain risks in news articles. The work of Zhao et al. (2024b) depicts a particularly high relation to the present research. Their introduced framework incorporates LLM-based risk identification in news articles and subsequent risk classification using the Cambridge Taxonomy of Business Risks (CTBR). This framework is demonstrated by means of a software prototype named LARD-SC (LLMs for Automated Risk Detection in Supply Chains). To enhance output accuracy and relevance, sophisticated prompt engineering is applied.

However, it becomes evident that none of the above-mentioned LLM-based approaches address operational parameter optimization as discussed in Section 1 portraying room for potential improvement and a clear research gap (cf.  Appendix).

Aiming to make an initial step into filling the above-highlighted research gap, the following research questions are derived:

As mentioned, the Doe approach as defined by Kleppmann (2020) is leveraged for a systematic exploration of the parameter space by varying those in a controlled and structured manner. This systematic variation supports in efficiently allocating resources by minimizing the number of experiments to be performed while providing meaningful insights into effects.

Through Doe, the impact of individual parameters and their interactions on classification precision for supply chain risk identification is analyzed on a quantitative basis. Doe aims for achieving a comprehensive understanding of the system under study, enhancing the reliability and robustness of the study results and potential allowing for the knowledge transfer to other application cases. Due to the missing research on the impact of LLM parameters in the light of the presented application in supply chain risk identification this approach has the potential to help resolve the knowledge gap and also to further the understanding regarding the impact of LLM-Parameters in other cases.

In Figure 6 the Doe process is outlined. The process starts by creating an understanding of the initial situation (1). This is essential to clearly define what knowledge is to be gained through the research. The research objectives can then be derived on this basis (2). Specific hypotheses may be drawn upon. These are to be placed in a targeted manner in the context of supply chain risk identification. The next step is to define how success can be measured in the present use case (3). As this is a field with substantial room for interpretation, this step is particularly relevant. In order to better understand the relationships between the input and target value(s) developed in the previous step, an analytical research plan is then developed (4). This step concludes the planning part of the Doe process. This is followed by the implementation of the research (5). The data is obtained by carrying out experiments and subsequently being analyzed. Based on this evaluation, the results are interpreted and measures are determined as to how they can be translated back into practice and lead to added value (6). The effectiveness of these developed measures is then validated to delineate potential anomalies in the test data for instance (7). Based on this reconciliation and validation, the data obtained via the process, as shown in the figure, can then be reintegrated back into the process and further investigations can be carried out on this new foundation (8).

The Doe approach itself requires that the LLM architecture be kept constant, and the focus on binary outputs controls potential unknown effects and enables a more isolated analysis of the causal effects of temperature and Top P, increasing the reproducibility of the Doe approach.

The analysis of the initial situation is the first phase of a DoE-Process and serves to precisely formulate the problem and to identify relevant variables and potential restrictions. Without a sound knowledge of the research context, there is a risk of inadequate operationalization or neglect of confounding factors. This step ensures that the research design is based on realistic assumptions and is methodologically and conceptually coherent with the existing body of knowledge in the light of the application of Doe.

This study is conducted within an overarching research project “WiReSt” ^[1]. The focus lays on improving supply chain resilience through a robust early warning system for risks (SCREWS). Here, the utilization of novel technologies such as LLMs for real time risk classification represents a core part as it enables to acquire and process vast amounts of quantitative and qualitative data that could unveil latent risks in early stages. Previous studies within this research project have already outlined first concepts and an initial framework for AI-based real time global risk assessment into which the outcomes of the present study will be assimilated (Eschenbächer et al.). Figure 7 displays the conceptual process of SCREWS from which real-world events and company specific information leads to different reports on potential sources of supply chain risks. The research on the early warning system is focused on the steps I–IV (marked blue). The last but most essential step lies in the data classification. The classification is carried out on a case-by-case basis, taking into consideration the individual circumstances of the company for which the evaluation is performed. Due to the integration of this way of evaluating the individual news the value of each alert is increased. This is also where the results of this work are applied.

Deriving precise research objectives and hypotheses is essential to focus on clear knowledge objectives. This step enables a hypothesis-driven approach that allows a structured examination of specific cause-and-effect relationships. In this context in particular, the formulation of research objectives helps to maximize the validity and relevance of the results by directly targeting existing research gaps.

The aim of this study is to increase the knowledge on how to adjust operational LLM parameters for supply chain risk identification (in this case correct classification weather the content of a news article is to be assessed as critical or not). There are three modifiable elements in the overall process at the meta-level. These are the input itself (e.g. a news article in combination with a prompt), the set variables and the LLM itself. The objective of the study, to evaluate the parameters, emerges from the tabular representation of the two factors consistency and workability (in the context of this study) in Figure 8. These two conditions have to be met for an examination to be relevant in the context of this study ^[2]. The objective of this study therefore includes investigating the effects that modifications to the variables have on supply chain risk identification. This aligns with the determined research gap in this context.

The identification of the target values (response variables) and the experimental factors is the central aspect of Doe. By systematically selecting the factors, main effects and interactions can be investigated, enabling a differentiated analysis. Insufficient specification of the variables, on the other hand, can lead to erroneous conclusions.

Next, the target values and the factors with a potential impact on these values are to be determined. As highlighted in the chapter on the methodological approach, the definition of the target variable is of particular interest. How the target variable is assessed in the context of this study is shown on the right-hand side of Figure 9. In the first instance, a distinction is made between two categories based on the output of the LLM. Firstly, there are responses that are in the expected format (either “0” for a non-critical or “1” for a potentially critical event) or those that are not (other outputs, e.g. words or other numbers). If the response is in the expected format, the correspondence with the target evaluation of the corresponding information is to be assessed as shown in Figure 9. For each article that is later used as input, the researchers assess whether it should be rated as critical or uncritical. This rating represents the value that the model should output.

In terms of factors, there are potentially seven operational parameters that could be examined as part of this study. To focus on the parameters with the greatest potential added value in the context of the use case at hand, a pre-selection was performed by the researchers. An exclusionary procedure was used, whereby variables that could be problematic in practical application were excluded and the remaining variables were further researched. The variables that are relevant to the research are marked with an “R” in the figure (Temperature and top P). The temperature parameter regulates the randomness of the model’s output, with higher values promoting increased diversity in generated responses (La Vega, 2023). On the other hand, the top P parameter controls the probability of distribution of potential next tokens by considering only the most probable ones, based on their cumulative probability mass (La Vega, 2023). These parameters play a crucial role in shaping the coherence, diversity, and quality of language generation in LLMs, influencing their overall efficacy in various natural language processing tasks (La Vega, 2023). The model selection was excluded due to the frequent version changes in the LLMs made available. The parameters of maximum length, frequency penalty and presence penalty were excluded from the analysis due to the expected output. As the evaluation should be either “0” or “1”, parameters that focus on the effect of values that have already been output are not expedient. Using the maximum length is not practicable due to the varying length of the input in the form of news articles. The use of the “best of …” parameter is potentially interesting, as various results are generated here and only the most suitable ^[3] result is output. However, this parameter is only available for legacy models.

The design of an experimental plan ensures that statistical efficiency is maximized, and experimental effort is minimized. In a scientific context, this step serves to systematically control disturbance variables and reduce measurement uncertainty, enabling more precise modelling and interpretation.

Figure 10 shows the experiment plan to be followed. The temperature parameter is changed first, ceteris paribus. This is due to the fact that it is likely to have a stronger influence on the results due to the direct impact on the next token; the results from this procedure are then subjected to analysis. Based on the potential optimum value determined from the analysis, the top P parameter is now adjusted ceteris paribus and the results are evaluated. In the third and final step, the effect of changing both parameters is to be examined based on the findings from test 1 and test 2. This is carried out using random samples to ensure stability of the previously determined optima.

The conduct of the experience according to the designed plan under controlled conditions is a critical step that ensures that the collected data is valid and reliable. The control of experimental conditions is necessary to ensure internal validity. It is essential to consider both replication and randomization to neutralize confounding variables and maximize the generalizability of the results.

To increase the reliability and validity of the findings, a large sample size is expedient, as, for instance, the temperature parameter inherently functions to amplify the randomness of responses with increasing values. 21 assessment objects are used, each of which occurs 41 times in each processing batch, to reduce the number of random effects. The examinations with a direct focus on the temperature parameter are carried out 5 times with the same values, resulting in a total number of 4,305 evaluations as basis for the calculation of mean values. As depicted in Figure 11, the modifiable variables are stored directly in a Microsoft Excel table used for the study and the evaluation results are then automatically transferred back into this table so that the table can be evaluated ^[4].

The evaluation of the experimental results is an integral part of Doe as it forms the basis for hypothesis testing and causal inference. Thus, making it possible to quantify main effects and interactions.

During this step, the data obtained in the previous steps is analyzed. This is carried out using Microsoft Power BI to enable automated evaluation of new data and analysis of individual effects in a large data set. The interim results are presented in the following sub-chapters.

Figure 12 shows the result for 5 tests carried out, each with 861 samples consisting of 21 questions in 0.05 increments for the temperature parameter in an interval of 0.00–2.00. The deviations show that the relative stability of the results begins to decrease for the range from a temperature value of 1.40 onwards. Optima in relation to desired results are present at both 0.25 and 0.60. Due to the greater distance to more unstable results, 0.25 is defined as the potential optimum value for the temperature parameter for fixed other variables.

Figure 13 shows an example of the data for the execution at two different prompts with variation of the temperature parameter in 5 test runs. This shows the high tendency of the LLM to output changing results measured against a deterministic output.

The influence of the top P parameter is evaluated in the same way as already described in Section 4.6.1. The evaluation shows that there are no significant fluctuations in the results up to a threshold value of around 0.5 (see Figure 14). After the threshold value is exceeded, the fluctuation between correct and incorrect categorizations increases. There are no non-valid inputs. Due to the fluctuations in both positive and negative directions, no reliable optimum can be determined for the top P value at this point.

In this case, the procedure is as described in 4.6.1 and 4.6.2. 41 combinations of random values in the ranges of 0.00–2.00 (temperature) and 0.00–1.00 (top P) are generated and fed into the process. To make the parameter–performance relationship easier to read, a simple naive Bayes classifier was fit to the experiment outcomes, using Temperature and Top P as features and the binary label correct vs. incorrect as the target. The curves plot the model’s estimated class-conditional likelihoods for correct vs. incorrect predictions over the alterations explored in the Doe. Regions where the correct-class curve sits above the incorrect-class curve indicate empirically more reliable settings. Figure 15 shows that the optimal value for temperature to support in leading to a valid assessment by the LLM is at around 1.1., the biggest positive distinction from wrong prediction is until the lines intersect at approx. 0.8. The graph compares the probabilities estimated by the model for correct (red) and incorrect predictions across parameter ranges. The increased probability for the correct class at medium temperatures (≈0.4–0.7; decreasing by ≈ 0.8) visualizes the stability band identified in the Doe, while Top P shows weaker, threshold-like effects.

Interpreting the results aids in laying the foundation to verify the results and also to derive practical implications from the gained knowledge later on. The interaction between the operational parameters, particularly Temperature and Top P, and the target variables provided critical insights into the configurations required for optimal classification performance. The Temperature parameter demonstrated a dominant influence on the model’s behavior, directly modulating the randomness inherent in the generated outputs. Configurations within the range of 0.4–0.7 (on a scale from 0 to 2, 0 meaning deterministic and no randomness while 2 is the maximum amount of randomness possible) allowed for an equilibrium between diversity and determinism, thereby facilitating consistent binary classification outcomes aligned with the study’s objectives. However, when the Temperature exceeded 0.8, the outputs became erratic, often deviating from the intended binary classifications, which suggests that excessive randomness compromises the coherence of the predictions and conflicting with a binary classification task.

In contrast, the Top P parameter exhibited a comparatively limited effect within the tested ranges. Accuracy variations became more pronounced only beyond a threshold of 0.5, but its overall contribution to classification precision remained marginal. This finding underscores the dominant role of the Temperature parameter in shaping model performance, particularly in applications requiring robust and high-fidelity outputs. When analyzed in combination, the interplay between Temperature and Top P reaffirmed the preeminence of Temperature as the critical determinant of output quality. Optimal results were achieved with a Temperature setting of 0.6, accompanied by a Top P value of 0.5, although the latter did not consistently enhance performance.

The verification process was undertaken to confirm the robustness, consistency, and validity of the experimental findings. Additional experiments using randomly selected input samples were conducted, reinforcing the observed parameter effects and validating the reproducibility of the results. These supplementary evaluations confirmed that the Temperature parameter exerts a substantial and predictable influence on classification accuracy, while the Top P parameter’s role is secondary and context-dependent within this specific use case.

The incorporation of randomly selected samples ensured that the observed parameter effects were not artifacts of specific input sets but instead reflected generalized patterns across diverse data contexts. By repeatedly testing these random samples under controlled experimental conditions, also with varying prompts and content and parameters, the observed results fit in the context of the derived conclusions of this work.

The outcomes of this research present practical implications for the deployment of Large Language Models (LLMs) in supply chain risk management, as well as for the broader domain of operational parameter optimization in AI-driven decision-making systems. For practitioners, particularly those addressing the complexities of risk classification, the findings underscore the critical importance of precise parameter calibration. Achieving reliable and interpretable outputs in dynamic and high-stakes scenarios necessitates a deep understanding of parameter behavior. The study’s systematic delineation of the optimal range for the Temperature parameter provides a robust framework for enhancing model efficacy by balancing the inherent trade-offs between randomness and determinism. This balance is essential for risk identification tasks, where even minor inaccuracies in classification can lead to significant operational or financial consequences. In public-interest supply chains (food, pharmaceuticals, medical devices) calibrated early warnings and the potential to reduce false alarms and missed incidents, stabilizing upstream planning and last-mile availability would yield a positive societal impact.

The identification of the Top P parameter’s limited influence within specific ranges further refines the approach to parameter tuning. This insight allows practitioners to prioritize configurations that yield the most substantial performance gains, streamlining the optimization process. By reducing computational demands and simplifying implementation, these findings make advanced LLM configuration strategies more accessible, even to those with limited expertise in machine learning.

Beyond the immediate application to supply chain risk management, this study offers a replicable and generalizable methodology for systematically optimizing LLM performance in diverse contexts. The Design of Experiments framework used herein ensures that the findings are not only statistically robust but also practically adaptable to a wide array of classification tasks, including fraud detection, customer sentiment analysis, and crisis management. This methodological adaptability highlights the scalability of the approach, underscoring its relevance to numerous domains where high accuracy and reliability are indispensable.

The study’s emphasis on iterative validation, with a particular focus on random sampling, further solidifies its practical contributions. Random sampling proved instrumental in verifying the consistency and robustness of the proposed parameter configurations, ensuring that the observed effects are generalizable across varied data conditions. This iterative and data-driven process minimizes the risks of overfitting while enhancing the external validity of the findings. The integration of such rigorous testing frameworks establishes a benchmark for deploying LLMs in environments characterized by variability and uncertainty.

In conclusion, the insights derived from this research extend far beyond the specific case of supply chain risk management as outlined in SCREWS. They provide a foundational approach for optimizing LLM configurations that is both systematic and adaptable, paving the way for further innovation in the deployment of AI-driven systems across complex and evolving operational landscapes. By coupling theoretical rigor with practical applicability, this study bridges the gap between academic research and real-world implementation, offering valuable tools for practitioners and researchers alike.

The aim of this research to increase the knowledge related to the impact of individual parameters and their interactions on classification precision for supply chain risk identification is to be considered achieved. By using the Doe process, a research design was set up in a structured manner which enabled findings against the background of the present use case and led to new insights on the performance of LLMs for supply chain risk identification. Finally, based on these findings, the initially derived research questions can be answered accordingly. Providing an answer to RQ1 (Which operational LLM parameters have the highest impact on the accuracy of supply chain risk classification based on external data?), it was found that the Temperature signifies the highest impact on the classification accuracy against the background of the given use case.

With regards to RQ2 (Which operational parameter configurations maximize the predictive accuracy of LLMs in supply chain risk classification, while ensuring robustness and adaptability across diverse operational scenarios?), it was found that the Temperature parameter significantly influences the model’s behavior, directly controlling the randomness of the generated outputs. Settings between 0.4 and 0.7 achieved a balance between diversity and determinism, enabling reliable binary classification results that aligned with the study’s goals. Marginal, positive accuracy variations through Top P became noticeable when the threshold exceeded 0.5. In combination, optimal results were achieved with a Temperature setting of 0.6, accompanied by a Top P value of 0.5.

In conclusion, this study underscores the pivotal role of operational parameter optimization in enhancing the efficacy of LLMs within the domain of supply chain risk classification. The findings emphasize the necessity of fine-tuning the Temperature and Top P to balance classification accuracy and output stability effectively. By leveraging a systematic Design of Experiments framework, this research has not only made an initial step into filling a critical gap but has also established a replicable, scalable methodology for optimizing LLM performance in this and related contexts.

However, certain limitations apply. It is to note, that a “randomness” variable is inherently not the intended use-case for the application of Doe. The randomness makes it more difficult to test the changes of other variables while maintaining a constant “randomness”. A high randomness is the same parameter but causes different outputs each time the process is initialized. This was mitigated by the high number of tests done.

Another key limitation lies in the binary nature of the classification task. While this approach allowed for focused evaluation of the input variables, it inherently limits the generalizability of the results to multi-class classification scenarios or more nuanced tasks that may involve multiple risk categories. Expanding the framework to encompass such complexities would enhance the practical relevance and applicability of the findings. Multi-class taxonomies for risk could also pose a viable alternative.

Finally, the experimental setup employed fixed LLM architectures without exploring the impact of different model architectures or sizes. Given the rapid evolution of LLM technologies, comparative analyses involving diverse architectures would add significant depth to the findings and extend their applicability.

Future research should seek to expand upon these contributions by addressing the identified limitations. Incorporating multi-class classification schemes, integrating more heterogeneous data sources, exploring a broader set of parameters, and systematically testing different LLM architectures will be essential for ensuring that the optimization framework remains both robust and versatile. This iterative process will further enable the refinement of AI-driven approaches, equipping practitioners and researchers with tools to navigate increasingly complex and dynamic operational environments with greater confidence and precision.

The paper is related to the project “Economic resilience in the Steinfurt district (WiReSt).” “The ‘WiReSt’ project is being implemented within the ‘Region gestalten’ program of the Federal Ministry of Housing, Urban Development, and Construction in cooperation with the Federal Institute for Research on Building, Urban, and Spatial Research. Das Vorhaben, WiReSt” wird innerhalb des Programms Region gestalten des Bundesministeriums für Wohnen, Stadtentwicklung und Bauwesen in Zusammenarbeit mit dem Bundesinstitut für Bau-, Stadt- und Raumforschung gefördert.

This paper forms part of a special section “Advancing Sustainable Connectivity: Innovations in Logistics and Supply Chain Integration (ISL 2024)”, guest edited by Ruth Banomyong and Kamrul Ahsan.