Deep reinforcement learning-based schedule optimization for parallel precast production Open Access

Precast components (PCs) refer to structural or architectural elements that are manufactured in a controlled factory environment before being transported to a construction site for installation. Utilizing PCs in construction offers several advantages over traditional cast-in-place construction methods (Jang and Lee, 2018; Anvari et al., 2016), including environmental sustainability (Ma et al., 2021), high quality, and on-site time savings (Mao et al., 2013; Tam et al., 2015). For a precast concrete project, the production of prefabricated components is a critical factor constraining the project’s completion timeline. As the PC market grows (Zhou and Ren, 2020), issues may arise due to the absence of production planning and scheduling, leading to early or delayed deliveries, excessive inventory, prolonged construction times, and increased expenses (Wang et al., 2018). Therefore, PC production scheduling (PCPS) is crucial for the PC industry.

Recently, research has been conducted to identify several practical concerns in real-world PC production. First, current practices primarily rely on classic dispatching rule (DP) methods (Kim et al., 2017; Lin et al., 2019). Earliest Due Date (EDD) and Shortest Process Time (SPT) are the most commonly utilized rules for managing a large number of production orders. While these DP methods can quickly generate a feasible plan, they lack flexibility handling diverse orders and cannot guarantee an optimal scheduling policy. DPs often oversimplify production objectives, which typically results in convergence to local optimum when considering multi-objective optimization. Second, human expertise continues to play a crucial role in determining detailed production plans (Du et al., 2020), especially regarding resource allocation across multiple production lines. Schedulers often can only provide estimated values for various resource needs and lack a clear understanding of the resource flow direction. This can lead to suboptimal utilization of resources and create challenges in achieving multiple optimization objectives. Third, there is an increasing demand for rapid rescheduling in response to parallel production system with real-world constraints (Wang et al., 2021), as order quantities and due dates are often variable. Schedulers need a constrained rescheduling method that can adapt to the diverse demands of different orders.

Many studies have been made to propose optimal precast production scheduling methods (Wang et al., 2023). Among current studies, heuristic algorithm especially Genetic Algorithms (GAs) are the most commonly used baseline algorithm (Peiris et al., 2023; Xie et al., 2024). Leu and Hwang (2001) were the first to applyGA with improved random crossover and mutation strategies to solve the resource-constrained production scheduling problem in PCPS. GA demonstrated flexibility in generating near-optimal solutions for the proposed flow shop model. After their work, GA was improved to further solve multi-objective schedule optimizations such as minimizing total makespan, penalties for earliness, and penalties for tardiness. Those improved GA outperformed the traditional DP methods in multi-objective tasks (Chan and Hu, 2002b; Dan et al., 2021). However, most research has focused on the idealized setting of a single production line (Du et al., 2020, 2021), while actual factory environments typically involve multiple parallel production lines to execute order manufacturing concurrently (Dan et al., 2021). Studies on optimization in parallel production are limited and their core algorithms predominantly rely on GA (Yang et al., 2016; Wang et al., 2021). As an Evolutionary Algorithm (EA), a key limitation of GA is its requirement for numerous iterations to generate an optimal schedule. The required number of iterations can significantly impact performance, with additional iterations leading to prolonged computational time. Moreover, the scheduling policies produced by the GAs lack the adaptability to changing initial conditions. For different PC orders, the GA needs to be recomputed to obtain rescheduling results. This means that GA cannot be used as a pre-calculated scheduling policy. In the complex and dynamic scenarios typical of real-world applications, the use of GAs presents significant challenges.

Compared with evolutionary algorithms, Deep Reinforcement Learning (DRL) shows superiority in its generalization ability and has been applied in a wide range of research domains (Sivamayil et al., 2023). A well-trained RL agent could even handle very complex tasks such as StarCraft II (Vinyals et al., 2019), Go Game (Schrittwieser et al., 2020) and flexible recommendation systems (Sivamayilvelan et al., 2024). In the field of PCPS, Du and Li (2024) integrated DRL with an EA to optimize parallel PCPS. They used DRL as an operator selector for evolutionary algorithms. The schedules produced by the EA required further processing through two additional refinement strategies. However, their method was essentially still an EA and algorithm iteration remains mandatory. Thus, their method cannot be used as a pre-trained rescheduling policy. Kim et al. (2022) applied DRL to directly make an optimal production plan with a single objective of minimizing the tardiness penalty. They designed an agent with the action space consisting of four classical DPs. However, their work did not discuss DRL performance on parallel production scenarios. Their method still relies on DPs to decide production sequences. The DRL policy can select only one job at each decision point, which cannot be directly implemented on parallel production scenarios.

Despite the growing interest in DRL for production scheduling, current studies lack a thorough investigation into parallel PCPS and the rescheduling capabilities of pre-trained DRL policies. The complexity of parallel production lies in the combinatorial explosion of potential job sequences across multiple production lines and diverse PC types. A major challenge in applying DRL as a pre-trained scheduling policy is enhancing its ability to generalize across varying production conditions. Generalization is critical for enabling DRL to generate feasible and optimized schedules in scenarios that differ from the original training configuration.

To address these practical demands and bridge the identified research gaps, this study proposes a DRL-based multi-objective scheduling framework that minimizes total makespan and earliness/tardiness penalties while incorporating real-world constraints such as mold plate availability, limited labor resources, and working hour restrictions. Unlike traditional methods that require retraining or recalculation for each new production scenario, the proposed approach develops a generalized policy capable of rapid and adaptive rescheduling. The framework is designed to support both initial schedule generation and responsive adjustments, offering a practical and scalable solution for modern precast production systems. The key contributions are summarized as follows:

1. A novel DRL architecture with an efficient and scalable action space design for parallel PCPS.

The proposed framework introduces a new action space formulation that supports the simultaneous selection of multiple jobs for parallel processing. As the number of PC types and production lines increases, the number of possible job combinations expands rapidly, which can hinder training efficiency due to the large parameter space. To mitigate this, the action space is strategically constrained, and an action masking mechanism is applied to dynamically mask invalid actions during training and inference. This approach significantly reduces computational complexity and enhances training convergence. As a result, production managers can generate optimized schedules quickly and make timely decisions on incoming PC orders.

1. A generalizable DRL model capable of effective rescheduling under varying production scenarios.

The state representation integrates both resource utilization and order progress, capturing the dynamic status of the production environment. A deep neural network, based on convolutional layers, is used to extract rich features from this composite state. Crucially, the state design is decoupled from specific job quantities and due dates, allowing the DRL model to generalize across different scheduling tasks. Once trained with fundamental facility parameters (e.g. mold plate availability and labor resources), the resulting pre-trained DRL policy can be directly applied to reschedule tasks with different order configurations—without the need for retraining. This enables practical deployment in real-world PC factories where order variations are frequent and rapid rescheduling is essential.

The rest of this paper is arranged as follows: Section 2 introduces the related works in the field of PCPS. Section 3 describes the problem definition and the optimization objectives. Section 4 illustrates the details of proposed DRL method. Section 5 conducts experiment on case studies. Section 6 makes discussions on the results and Section 7 analyses the limitation. Finally, Section 8 makes the conclusion of this study. The key abbreviations of this paper are listed in  Appendix 1.

PCPS is an NP-hard problem due to its complexity, combinatorial nature, the presence of discrete variables, and the lack of a polynomial-time algorithm that can solve the problem exactly (Du and Leung, 1990; Lin and Liao, 2013). Researchers have conducted studies on precast scheduling optimization over the past decades. Several optimization theories have been adopted to illustrate the PCPS problem. Dawood (1995) proposed a simulation-based method and defined the PCPS problem as a job shop scheduling problem. Chan and Hu (2001) chose the Flow Shop Scheduling Problem to identify the relationships between jobs and machines in the production processes. Given these diverse problem formulations, the optimization algorithms used are also varied. Chan and Hu (2002a) adopted a Constrained Programming (CP) approach and their research showed CP outperformed traditional heuristic rules in makespan and cost. Khalili and Chua (2014) introduced a mixed integer linear programming and the model reduced the production cost by over 9%. Apart from these algorithms, hybrid methods that combine multiple algorithms were proposed. Metaheuristic algorithms are also widely used in the field of project management (Alshboul et al., 2025a, b). Chang and Han (2021) improved a discrete differential evolution algorithm with a local search strategy. Xiong et al. (2021) proposed a hybrid tabu search and iterated greedy algorithm and compared it with a hybrid GA and variable neighborhood search and a two-phase heuristic method. The computational analysis indicated that the proposed hybrid method performed better than others on average. Du et al., (2023) used a biogeography-based optimization algorithm to schedule PC lean production. However, these studies did not address parallel production or rescheduling capability.

Apart from these diverse optimization algorithms, GA is more widely adopted as it provides better optimization in off-site construction and PCPS (Li and Love, 1998). Leu and Hwang (2001) first used GA to solve the resource-constrained production scheduling problem in PCPS. They adopted improved random chromosome operators for gene insertion and mutation, which outperformed previous algorithms in optimizing makespan. GA showed flexibility in generating near-optimal solutions for the proposed flow shop model. In their later research (Leu and Hwang, 2002), an improved union crossover operator was employed to accommodate the random key representation. Chan and Hu (Dan et al., 2021) first proposed a normalized GA to optimize multiple objectives. Inspired by their research, the most frequently multi-objectives include minimizing makespan as well as earliness and tardiness penalties (Chan and Hu, 2002b). Other major concerns in multi-objective PCPS research include the cost of mold type change (Benjaoran and Dawood, 2006), minimizing machine idle time (Liu et al., 2023) , and minimizing total labor over time (Liu et al., 2021) are also major concerns in multi-objectives PCPS research. Dan and Liu (2024) proposed a GA based integrated scheduling method for production and transportation, which aimed at minimizing earliness/tardiness penalties and transportation cost under delivery time window. Mao et al., (2024) used a hybrid GA for just-in-time delivery of PCs. Xie et al., (2025) improved GA with topological sorting priority method to optimize production makespan. Their work presented superiorty compared to other optimization methods.

In terms of rescheduling, Ko (2010) utilized a schedule adjustment principle to address the issue of demand variability based on customer behaviors, although their approach lacked a robust optimization mechanism. Kim et al. (2020) introduced a discrete-time simulation method to manage due date uncertainties. Du et al. (Ko and Wang, 2011) proposed an elite GA to address demand fluctuation. Later, they proposed a multi-agent model with GA to establish a dynamic decision support framework. However, their study was limited to a single production line. In terms of parallel production with multiple production lines, Wang and Hu (2018) utilized GA to dynamically reschedule in response to frequent demand fluctuations across different types of PCs. Wang et al., (2021) developed a hybrid GA method aimed at improving machine efficiency and mold utilization during machine breakdowns. Nevertheless, as an iteration-based metaheuristic, GA requires reinitialization of configurations when making reschedule, which means it cannot be used as a pre-trained scheduling policy. Fixed parameters, such as the mutation rate, do not dynamically adapt to different rescheduling scenarios, potentially leading GA to converge on local optima in rescheduling optimization tasks.

DRL is an area of artificial intelligence that is initially inspired by the learning abilities of animals in nature. It simulates the response of the animal, namely the agent, to the received environmental feedback reward after taking an action (Figure 1 (Yao et al., 2024)). In DRL methods, the agent aims to maximize the total reward by interacting with the dynamic environment through making optimal decisions (Sutton and Barto, 2018). Unlike traditional supervised and unsupervised learning, DRL algorithms do not require external data for the training and validation (Kaelbling et al., 1996; Kurinov et al., 2020). The agent automatically collects data and feedback from the simulation environment and eventually reaches an optimum action policy. Moreover, RL methods show superiority in the area of sequential decision-making (Moerland et al., 2020). The implementation of DNN makes RL methods capable of learning from high-dimensional inputs, which means DRL can solve real-world complexity (Mnih et al., 2015).

The application of DRL in production scheduling can be categorized into two types: (1) directly generating optimized production schedules through interaction with the environment from starting to ending; and (2) using DRL as heuristic operator to refine an initial production schedule. The first approach is designed to derive a policy that facilitates end-to-end production scheduling. Similar research has been conducted in the field of job shop scheduling to optimize production problems with real-world complexity (Li et al., 2022; Pan et al., 2021; Luo et al., 2021; Du et al., 2022b). The second approach typically requires DRL to be integrated with heuristic algorithms (Chen et al., 2020; Lin et al., 2022; Du et al., 2022a).

With DRL application in related field, the research on precast production schedule optimization remains limited. For the first approach, Kim et al. (2022) proposed a Deep Q-Network (DQN) method to generate precast production schedules with minimum tardiness penalties. They adopted four DPs as DRL actions to interact with their precast environment. Their results indicated that using DRL for end-to-end schedule optimization outperformed other classic DPs. However, their method still relied on DPs as actions to schedule PC orders. Parallel production was not addressed in their case studies, as their action design only supported selecting one job at each decision point. In contrast, real-world parallel production often requires the execution of multiple jobs simultaneously. Their DRL architecture also lacked generalization ability as their state design only focused on the progress of PC orders. Resource utilization was not adequately considered in their DRL framework. For second approach, Du and Li (2024) further discussed the DRL integrated heuristic method application in distributed precast production. They designed 17 evolution operators as actions for algorithm iteration and refinement. However, both studies have limitations in considering various constraints. Kim et al. (2022) set limitations to the amount of molds, but they did not set same limitations to the labors. Du and Li (2024) considers only time-of-use electricity price as the constraints. Their work did not take other typical resources constraints or labor constraints into account. The omission of these key constraints renders their approaches less consistent with actual production conditions, highlighting a major limitation of their studies.

Table 1 presents a summary for the most relevant studies with details of their methodologies based on the literature review. GA methods can address the PCPS problem, accommodating various constraints and multiple objectives. Nonetheless, when rescheduling tasks become necessary, GA requires recalculation with new initial configurations for the rescheduling tasks, as GA only provides optimized schedules but not a policy for scheduling. Current DRL based methods also have limitations. Kim et al. (2020) considered only a single objective and did not consider labor constraint. Their work was not capable of optimizing parallel PCPS. Dy and Li (2024) did not explore the use of pre-trained DRL as a rescheduling policy in parallel production, as their method still required an EA for schedule optimization. To bridge the gaps, this study proposes a multi-objective DRL method for parallel production. The proposed DRL is capable of generating a pre-trained policy suitable for task rescheduling.

This study categorizes PC production as a fixed-location production system, which requires fewer material and labor movements (Yang and Lu, 2023). Contemporary PC factories adhere to a procedure where concrete preparation precedes the production process, and it is subsequently transported via specialized cranes to maintain a consistent supply. Additionally, reinforcement is pre-manufactured in advance. Labor resources are also a critical factor, as specific work crews in the factory are responsible only for designated production processes, and crews do not interchange between tasks. Therefore, when creating production schedules, it is necessary to assess the impact of varying crew quantities on production progress. As revealed by the findings of the field study, the PC production process identified in this research comprises six distinct processes: mold assembling (P1), reinforcement setting (P2), concrete casting (P3), curing (P4), mold stripping (P5), and finishing (P6). These processes are graphically represented in the following figure (Figure 2), providing a comprehensive model of the production process from a flow view. In this study, each of the six processes is assigned to a dedicated crew. Upon completing their respective processes, these crews transit to a new mold plate and commence new tasks. When preceding processes are accomplished, the subsequent processes are undertaken by new crews. In essence, processes unfold on the same mold plate, with labor transitioning between different mold plates.

As indicated in the research of Ko and Wang (2010), certain PC production processes can be interrupted. The need to define interruptible processes arises from the fact that crews can only work a limited number of hours within a single workday. Once their work hours are exhausted, no new jobs are assigned to the crews. Interruptible processes can be paused after regular work hours and resumed at the beginning of the next workday. In contrast, non-interruptible processes must be completed during overtime hours and cannot be deferred to the following day. The interruptible processes include P1, P2, P5, and P6. Hence, P3 is an uninterruptible process because the casting process must be continuous. Furthermore, P4 is a unique uninterruptible process as it does not require crew assignment and must be performed consistently.

To sum up, the problem studied in this paper can be defined as follows: consider a PC factory equipped with $Nm$ mold plates, each of which is assigned to various groups of PC tasks. Each task comprises of $j$ types of different PC jobs that enters the mold plate at different times. When a job initially enters a mold plate, the crew responsible for P1 is assigned to that mold plate. If no crew is available at that moment, no new job will be allocated. In this study, even jobs that have already been scheduled on the manufacturing process and have completed some subprocesses might still be in the queue, waiting for available crews who are currently engaged in executing other jobs to be released (Figure 3).

In a typical PCPS, time and cost are the two major concerns. Managing these factors effectively becomes crucial for optimizing the production workflow and ensuring timely deliveries. The objectives of this research then consist of two parts: minimizing the total makespan and the penalties for earliness and tardiness. The relationships among makespan, tardiness, and due dates are illustrated in Figure 4. A reduced makespan not only indicates improved productivity but also ensures that resources are utilized efficiently, ultimately contributing to cost savings and enhanced profitability. Early completion of jobs can be as detrimental as tardiness, especially if it leads to excess inventory costs or disrupts subsequent processes. On the other hand, tardiness in delivering products can result in dissatisfied customers, financial penalties, and damage to the company’s reputation. In fact, earliness and tardiness penalties are conflicting performance indicators. Completing orders too early can lead to increased storage costs, while delaying production, although reducing storage expenses, may result in penalties for breach of contract. Therefore, achieving a balance between these two penalties is essential. The definition of the two objectives is presented in Equations (1) and (2) respectively. The variable explanations are provided in Table 2.

Objectives:

In the production of precast concrete elements, production efficiency is constrained by various factors (Wang et al., 2023). Researchers have identified many types of constraints in existing literatures. In this study, three types of constraints are considered to model the production problem, namely working hour constraints, labor constraints, and resource constraints. Working hour constraints refer to that the production jobs must be processed within specific time during a day (Podolski, 2022; Dan et al., 2021; Kim et al., 2022). Labor constraints refer to the availability and skill level of the workforce involved in the production of precast concrete elements (Chang and Han, 2021; Podolski, 2022). A limited number of skilled workers mean that there is a finite labor force available to handle the jobs at hand. Resources constraints refer to the availability of raw materials such as concrete, molds, and rebars (Yang et al., 2016). In this study, the quantities of concrete and rebar are assumed to be unlimited. The quantity of mold plates is the primary resource constraint. Production scheduling must align with plates' availability, ensuring that the production pipeline is optimized to utilize these resources efficiently. The descriptions of three constraints are given in Equation (3) and (7) respectively and the variables are presented in Table 3.

Subject to:

To simulate the PC production processes, several necessary assumptions have been made. These assumptions guarantee that the simulation of the production process reflects the real production situation as accurately as possible, while appropriate simplifications allow the problem to be modeled and solved more effectively.

1. Each production step is operated by specialized crews.

2. Crews can only transfer to next mold plate after their current step completed.

3. Production steps are non-interruptible.

4. There is no transfer time for crews between different mold plates.

5. There is no idle time between different production steps.

6. Concrete casting on different mold plates can be performed simultaneously.

7. The quantities of crews and mold plates are limited but renewable.

8. Mold plates are unordered, meaning that selecting a specific mold plate does not impact the execution of the order.

The proposed DRL architecture is depicted in Figure 5. The original PC order information includes quantities and due dates for each type of PC elements The initial PC order data, which includes the quantities and due dates for each type of PC element, is combined with information about the PC facilities, such as the number of molds and crews, to set up the production environment. Meanwhile, an initial action space that includes all possible job combinations is then established regarding the input information. The size of the initial action space is determined only by the number of mold plates and types of PC elements. The production environment outputs the current state, including mold utilization, crew utilization, order progress, and ongoing job progress. This state is processed alongside the initial action space through a valid action masking algorithm, which filters out any actions that are not executable under the current state conditions. In this study, DQN is chosen as the backbone algorithm. As the state $St$ is designed as a two-dimensional matrix, so it is appropriate to use a two-layer convolution neural network Convolutional Neural Network (CNN) which outputs a value $Qt(St,at)$⁠. For each possible action $at$⁠. Then the value $Qt$ will be refined as $Qut$ for valid actions $aut$⁠. The action with the highest Q-value is selected to be executed at that decision point. The chosen action represents the job that the DRL has selected to perform at that moment. Subsequently, the selected job is added to the execution sequence, and the environment state is updated accordingly for the next decision point. The proposed DRL architecture differs from previous research (2022) in three main aspects: (1) to enhance the generalization ability, this study uses CNN to extract hidden features from the state tensor rather than using scalar values; (2) this study considers the complexity of parallel production lines and PC types in action design, allowing for arrange multiple jobs to be arranged and processed simultaneously; and (3) this study adopted a reward shaping mechanism for optimizing multiple objectives. The key variables used in the DRL model are defined and explained in Table 4. The neural network diagram is presented in Figure 6. The agent-environment flow chart is presented in  Appendix 2.

To train a valid DRL agent, the state, action, reward, and environment need to be designed. In reinforcement learning, the environment refers to the external system or context with which an agent interacts and learns through feedback in the form of rewards, shaping the agent’s decision-making process. In this study, such interaction is simulated using Python. Another role of the environment is to identify validate executable actions at each decision point and provide this information to the agent, a process known as action masking, which will be explained in the next section.

In DRL, the final policy is derived from the Markov Decision Process, so the actions of the agent are determined solely by the current state. For the PCPS problem, the state needs to represent the details of the current production environment, including manpower status, mold plate utilization, overall job progress, and so on. Additionally, the state should exhibit distinct and clear transitions for different actions. This is because when state changes are not apparent, the agent struggles to extract sufficient features from subtle variations, making it difficult for DRL training to converge.

To meet the actual concerns in PC facilities, this study designed a mixed state $St$ to combine various types pf facility and production line information. $sm$ represents the information about the job currently being executed on the $mth$ mold plates.

Action space $Aˆ$ is a set of all the possible activity combinations. Instead of taking different DPs as actions Kim et al. (2022), in this study, the design of the action space primarily considers two aspects: the number of mold plates and the quantities of PC component types. The state transition when selecting an action is explained in  Appendix 3. For a batch of PC orders consisting of $j$ different types, a common approach in designing the action space is as follows: each mold plate can have j possible job allocations; therefore, in the case of $m$ mold plates, the size of the action space becomes $jm$⁠. However, when the number of PC types in the orders is large and the number of mold plates is significant, the size of the action space becomes exceedingly large. This combinatorial explosion makes it difficult for the agent to converge during training.

This paper introduces a practical method for representing the action space, significantly reducing the search complexity for the agent during the training process. Instead of designing the action from a simplistic random allocation perspective, this study comprehensively utilizes two types of information: the total number of mold plates and the variety of job types. The action vector is defined using the following formula Equation (9) and (10):

The job selection subjects to the total quantity of mold plates $m$⁠. Thus, the size of the action space becomes Equation.(11) which is smaller than $jm$ when $j$ and $m$ are large:

At each time step, the DRL agent chooses action $at$ from a fixed action space $Aˆ$⁠. This selection method is known as behavior policy. In the adopted DQN method, an $ε−greedy$ policy is applied (Equation.12).

The $ε−greedy$ policy tends to guide the agent to select the action which gives the maximum reward in the time step. At the beginning of the agent training, $ε$ is set to be a larger value to let the agent explore the action space in a random pattern. This policy prevents the agent to be trapped in local optimization and helps the neural networks have better generalization ability.

However, even within a narrowed action space, certain actions may still lead to constraint violations. Typically, such violations result in an early stop of the training episode, thereby requiring the agent to undergo more episodes to converge, consequently diminishing training efficiency. To address this issue, action masking (Huang and Ontañón, 2020) is employed. At each timestep, the mask provides a set of legally safe actions for the DQN agent based on the environmental state $St$⁠.

To mask the invalid actions, a masking function is introduced (Equation.13):

The masked action set $Aφˆ(St)={at|φ(St,at)=1}$ is defined for the environment state $St$⁠. After the action masking, the agent will only choose valid actions from $Aφˆ$ under the $ε−greedy$ policy in Equation.(12).

A reward is the feedback obtained by the agent after interacting with the environment. The definition of the reward function determines the optimization direction of the DQN model. In this study, reward shaping is adopted to enhance the training efficiency (Hu et al., 2020). The reward consists of three parts: reward that marks the completion of PC orders $RW1$⁠, reward that indicates makespan $RW2$⁠, and reward that indicates earliness and tardiness $RW3$ (Equation.(14) - Equation.(17)).

Equation.(14) represents the end reward for completing one episode. Equation.(15) represents a penalty mechanism where $k$ denotes a negative real number. This negative reward aims to shorten the total makespan by implying negative rewards. This reward encourages the agent to finish the scheduling completely as early as possible. The value of $k$ is determined by a trial-and-error process. In this study, integers between $[−50,0]$ were tested, and the results indicated that training efficiency was optimal when when $k=−20$⁠. It should be noted that the value of $k$ is not determined and can be vary for different cases. The trial-and-error approach is widely accepted in reinforcement learning studies (Silver et al., 2021; Booth et al., 2023). Equation.(16) indicates the reward for earliness and tardiness of the completed job in the timestep. This reward helps the agent to better balance the two opposite penalties. Equation.(17) represents the total reward.

The cases in this research are derived from field studies conducted at PC facilities in Jiangxi Province, China. Essentially, these PC facilities operate under a make-to-order production model, indicating that production only commences once the corresponding orders are received. The range of prefabricated components produced by these PC facilities is diverse, including floor slabs, wall panels, stairs, and external facade decorative panels, among other PC components. The surveyed PC facilities have both fixed production areas and flow shop production areas. Fixed production areas are typically used for manufacturing medium-sized PC components, whereas flow shop areas are mainly dedicated to producing larger structural elements. This research focuses on fixed-location production as primary case study.

The scheduling tasks in this study are based on the real PC orders that provided by the surveyed factory. The basic configurations of the PC facility are displayed in Table 5 and Table 6. In Case study 1, three distinct types of PC components (namely wall panel 0, wall panel 1, and roof panel 2) are produced in various quantities. The facility contains six fixed mold plates, and the daily working time is set to 8 hours, leaving 16 non-working hours. The penalty factors for earliness and tardiness (⁠$α$ and $β$ in Equation (16)) are set to −0.5 and −10 respectively according to the real contracts.

Given that the surveyed facility possesses a large open-air space for storing orders completed ahead of schedule, the penalty for earliness is comparatively less severe than the penalty incurred for breach of contract resulting from delayed production. In the factory, the number of specialized skill crews is limited, which means that in some cases, even with sufficient workstations, some PC orders cannot be executed due to insufficient labor availability. In such cases, it becomes essential to carefully schedule the execution sequence of different order types to ensure resource efficiency. In this study, each PC production process is handled by a distinct crew, and the number of crews is diverse and restricted. The order specifics that provided by the PC facility are outlined in Table 7.

The ability of the pre-trained agent to effectively manage different upcoming PC orders plays a pivotal role in ensuring the seamless operation of manufacturing processes. To assess the agent’s robustness and adaptability under diverse order conditions, Task 1 is used to train the DQN model, while two rescheduling tasks, namely Rescheduling Task 2 and Rescheduling Task 3 (as detailed in Table 7), are adopted and implemented for evaluation. Two rescheduling tasks present more intricate challenges to the agent. In both cases, the due dates and job quantities differ significantly from those in the original task. This variation reflects the complexity often encountered in practical manufacturing environments, where projects may vary significantly in scope and urgency. Such dynamic changes demand an agent capable of rapidly adapting its scheduling strategies to meet the unique demands of each order, ensuring timely delivery and optimal resource utilization.

It is worth emphasizing that in surveyed factories, most incoming orders predominantly revolve around specific types of PC components. This trend underscores the critical importance of the agent’s proficiency in efficiently scheduling future orders of recurring PC component type. The rescheduling tasks provided by the surveyed facility, differing in due dates and job quantities, reflect the real challenges faced in the industry. The fluctuations in due dates, a common occurrence in real-world production, necessitate an agent that can make real-time adjustments to its scheduling algorithms. As off-site construction heavily relies on a multitude of unpredictable factors, the pre-trained agent’s ability to dynamically respond to shifting timelines is important. Simultaneously, the quantities of PC orders are intricately related to the scale and complexity of the corresponding projects. Hence, the agent’s ability to handle different order quantities is equally important.

In addition to the large PC components, the surveyed factory also kept production records for some smaller prefabricated components. These smaller components typically require only a few workers and can be completed within a shorter period. However, they are subject to more stringent delivery requirements, often within one week. Therefore, careful scheduling of the production sequence is necessary to avoid incurring substantial penalties for breach of contract. Table 8 presents a portion of the order, which includes 10 different PC elements intended for the construction of prefabricated townhouses. PC0–PC3 refer to decorate panels, PC4–PC7 refer to thin wall panels, and PC8–PC9 refer to thin roof panels. Additionally, two rescheduling tasks are also provided, each with different quantities and due dates compared to the original order. Similar to case study 1, Task 4 is used to train the DQN model, while Task 5 and Task 6 are the rescheduling tasks with different quantities and due dates.

The proposed DQN consists of a two-layer CNN to extract features from the state $St$⁠. The proposed CNN architecture comprises two convolutional layers followed by a fully connected layer. The first convolutional layer, with a $3×3$ kernel and ReLU activation, processes input data to produce 16 output channels, followed by max-pooling. The second convolutional layer takes the 16-channel output, applies a similar operation with a $3×3$ kernel, and produces 32 output channels. ReLU activation and max-pooling are again applied. The final layer consists of two linear layers: the first with ReLU activation and the second with Softmax activation. These layers transform the output into a tensor representing the Q-values of all actions. In the DQN framework, the input state tensor dimensions are 10 × 6 for case study 1 and 17 × 6 for case study 2, representing different configurations of production status information. The output layer produces a Q-value vector of length 83 for case study 1 and 559 for case study 2, respectively, corresponding to the number of actions in each case. To train the DQN agent, the batch size is set to 256, update frequency is set to 3, with a 10,000-replay buffer size. The training was carried out with $ε$-greedy policy at an exploration factor of 0.95 and linearly decreasing to 0.01 at the end of the episode. Adam optimizer is adopted, and L2 Normalization is introduced during the network training. The hyperparameters are determined by Grid Search. The initial search space for hyperparameters was based on recommended values from the open-source project Stable Baselines3 (Raffin et al., 2021). The final hyperparameters were determined as presented in the manuscript. Through the model training upon different case studies, the DQN provides optimal convergences. The training data was derived from real-world production scenarios collected during field visits to PC factories. Tasks 1 and 4 were used to train the DQN model, while other tasks were reserved for testing the generalization capability. Table 7 and Table 8 present the specific task configurations used in training and evaluation. The DQN model is trained on a personal computer with an i7-11700 k chip and an RTX 3080. The interpreter is based on Python 3.9.0, PyTorch 1.12.1, and Numpy 1.23.4. Task 1 and Task 4 are used to train the DQN agent, while the other tasks are rescheduling task using the pre-trained agent. The parameter settings are listed in  Appendix 4.

The results were compared with traditional scheduling methods, including two classic DPs and a simulation-based GA. Selected DPs are EDD and SPT. EDD prioritizes jobs based on the EDD, while SPT prioritizes those with the shortest processing time. Due to the different cases and assumptions made in current research, existing algorithm in the field of PCPS cannot be directly applied in this study. To tackle the gap, a simulation-based GA is proposed in this study. The simulation-based GA consists of a modified GA (Liu et al., 2020) and a simulation environment. Adopted baseline GA has been validated by standard benchmark problems of size J30, J60, and J120 from Project Scheduling Problem Library, which is a well-known scheduling problem library created by Kolisch (1997). It has also been used for comparison in recent published papers (Hua et al., 2022; Peng and Zheng, 2023; Martin et al., 2024). In this study, each gene represents a PC element, with the order of the genes indicating the sequence in which these elements are arranged on the mold plates for execution. Given that the research focus is on parallel production problems, the simulation environment proposed here serves as the chromosome decoder for the GA. The simulation decoder ensures that at every moment, resources and mold plates are fully utilized. The adopted algorithm uses regret-based biased random sampling and serial schedule generation scheme to generate initial population. Evolution strategy consists of elite selection mechanism, two-point crossover, swap mutation, and linear decreasing probability-based mutation. The parameter settings of adopted GA are shown in Table 9. For the EDD and SPT DPs, the generation of schedules also relies on the simulation decoder, which directly decodes job sequences that conform to the EDD and SPT rules.

The training of DQN is illustrated in Figure 7–9. These figures display the smoothed reward curve, indicating that the agent stabilizes around the 90th episode. After this point, continued training does not significantly improve the agent’s scheduling performance and training reward. Therefore, it can be concluded that the DQN has achieved the training objectives at this stage.

Figure 8 illustrates the tardiness and earliness curves during the DQN training process. As the training progresses, the penalties associated with earliness and tardiness exhibit distinct trends. Specifically, while the earliness penalty increases, the tardiness penalty gradually decreases. Additionally, Figure 8 presents the evolution of the makespan and the sum of the two penalties throughout the training. As training nears completion, both the makespan and the total penalty converge toward optimal values, demonstrating the agent’s ability to balance timely completion with efficient resource utilization.

Table 10 presents the scheduling and rescheduling outcomes of both traditional methods and the proposed DQN approach. EDD method is chosen as the baseline as its wide usage in the surveyed real-world factories. The gaps of other methods with the baseline are calculated by Equation (21). Where $pi$ represents the results of three other methods and $pedd$ represents the results of EDD.

In Task 1, the results demonstrate that DQN achieves the optimal schedule with reduced total penalties and a shorter makespan. EDD schedules the task with smallest earliness penalty, but the makespan is longer than DQN method. Compared to DQN, GA provides a better earliness penalties; however, it incurs higher tardiness penalties. Among the four evaluated methods, SPT exhibits the poorest performance, with the highest total penalty. The CPU times of the methods show that although GA is capable of generating suboptimal solutions superior to traditional DPs, its processing time is significantly longer than that of the DQN model.

In both rescheduling tasks, DQN and GA outperform traditional methods (EDD and SPT) in terms of tardiness penalty reduction and makespan minimization. In Task 2, GA achieves the lowest earliness penalty, yet its tardiness and total penalties are larger than DQN results. In Task 3, DQN outperforms GA with less total penalty. And the process times of GA remains longer than the pre-trained DQN model. With apparent shorter time, the same pre-trained DQN can generate near-optimal solutions under different task settings.

Table 11 presents the findings from case study 2. In Task 4, the DQN method produces the most efficient schedule, characterized by the lowest total penalty, shortest makespan, and reduced CPU time. Compared to the EDD method, the DQN achieves a 15.8% reduction in total penalty and a 14% reduction in makespan, while also necessitating less CPU time. Regarding the two rescheduling tasks, the pre-trained DQN continues to outperform both traditional DPs and the GA in Task 5. In Task 6, although DQN does not generate the best schedule among the four methods, it still produces a schedule comparable to GA in the least amount of time.

To enhance transparency and stakeholder acceptance, this study provides visual and quantitative illustrations of the model validation process through comparative analyses across multiple real-world-inspired tasks. The validation was carried out by applying the pre-trained DQN model to both the original training scenarios and a series of rescheduling tasks with varying order quantities and due dates. As shown in Table 10 and Table 11, the DQN’s performance was benchmarked against traditional DPs (EDD and SPT) and a simulation-based GA. The results demonstrate the model’s ability to generalize beyond the training task, maintaining competitive performance in minimizing penalties and makespan across different task configurations. Detailed performance curves (Figures 7–9) further illustrate the convergence behavior of the model, showing how the agent progressively balances earliness, tardiness, and makespan during training. These multi-level validations strengthen the credibility of the proposed approach and confirm its practical value for real-world precast production environments.

The results obtained in both training and rescheduling tasks illustrate the strengths and implications of the proposed DRL-based method. The use of a DQN framework with a simplified action space enables the agent to make decisions that are both optimized and computationally efficient. Unlike traditional algorithms that require full recomputation for every new task, the pre-trained DRL policy leverages learned scheduling strategies, allowing for rapid rescheduling across orders with different due dates and quantities. This adaptability is particularly valuable in real-world factory environments where conditions frequently change. The integration of action masking further improves efficiency by masking infeasible actions, enabling the model to focus on valid scheduling paths and contributing to shorter convergence times and lower penalty costs. These results highlight the effectiveness of the DRL approach not only in generating optimized schedules but also in delivering practical benefits such as speed, flexibility, and reusability.

The performance of the proposed DQN method is compared with a simulation-based GA and two established classic DPs. The evaluation is conducted across three dimensions: (1) total penalties; (2) makespan; and (3) CPU time. The penalty and makespan serve as indicators of the algorithm’s effectiveness, while CPU time reflects its efficiency. The results demonstrate that the DQN consistently generated the most optimal production schedules on the tasks for training (Task 1 and Task 4), achieving lower total penalties and makespan. In the rescheduling tasks using the pre-trained DQN agents (Task 2, Task 3, Task 5, and Task 6), the DQN outperforms the baseline in producing superior production schedules. Such generalization ability comes from the state and action design. In this study, the state representation does not directly encode the specific quantities and due dates of the PC elements. Instead, the state focuses solely on the real-time job progress on the fixed mold plates. Therefore, when rescheduling orders of the same type, the pre-trained DQN agent is not affected by order quantities and deadlines. Similarly, the size and structure of the action space are not linked to the specific quantities of PC elements, so the DQN agent’s action selection remains stable across varying order volumes. However, such design has limitations under complex rescheduling orders and diverse due dates such as Task 6. Thus, the performance of DQN is slightly inferior to that of GA.

Although the DQN does not produce a better schedule than GA for Task 6, its outcome remains better than the baseline heuristics. This phenomenon can be attributed to the differences in optimization mechanisms, adaptability, and generalization capability between the two methods. Specifically, GA employs a population-based global search with inherent exploration and diversification mechanisms, enabling it to better adapt to task-specific penalty structures—such as the dominance of tardiness costs observed in Task 6—by effectively minimizing total penalties. In contrast, the pre-trained DQN relies on value functions learned from the training environment, which may limit its adaptability when facing states that deviate from the training distribution. While this may result in slightly higher penalties compared to GA, DQN still achieves the shortest makespan and significantly outperforms GA in computational efficiency, demonstrating its scalability and suitability for real-time rescheduling. Importantly, the rescheduling results further show that DQN is capable of dynamically adapting to changes in precast element quantities and due dates, highlighting its practical value in flexible and responsive production-construction coordination.

From a CPU time perspective, the DQN outperforms both classic DPs and GA in schedule generation time. DQN’s fast generation comes from the pre-trained policy, which enables the DQN to directly select PC elements with the highest Q-values without iterative computation. Meanwhile, the proposed simulation-based GA outperforms the baseline in all six task scenarios, suggesting that the improved GA can be effectively applied to parallel production scheduling. However, a notable limitation of GA lies in its inability to be reused across different tasks without re-iteration. Each new scheduling problem requires the GA to undergo a full optimization cycle, resulting in significantly longer CPU times compared to both DQN and dispatching-rule-based approaches. In two case studies, there are clear gaps in the schedule generation time. There are two reasons for such time diversity. First, in the case study 1, the process times for each type of PC elements are recorded as integer. Consequently, it’s enough to set each timestep in the simulation to equal one real hour. In contrast, for case study 2, where the recorded process times are precise to 0.1 hour, the simulation must be accordingly configured to reflect this finer granularity. A comparative analysis of the schedules from both cases reveals that the number of steps required in the simulation environment for the first case is significantly fewer than those needed for the second case. As a result, the CPU time to generate optimal schedules for the second case will be longer. Second, when the simulation environment functions as a chromosome decoder, it necessitates additional computational costs. At each timestep, it must search the entire action space and select actions that align with the gene sequence. Since the case study 1 has fewer element types compared to case study 2, and the two DPs used in this study need the simulation to decode and generate the final schedule, an increase in the variety of elements leads to a larger action space, which lowers the schedule generation efficiency. However, pre-trained DQN agent does not need to search the entire action space, the neural network could select the action with highest Q-value. Consequently, DQN require less time to generate optimal schedule plans.

The schedules generated by the DPs and proposed methods are presented in Table 12. The numbers in each sequence represent the code of the PC element in the order. For the task with same PC types, the job sequence given by EDD changes according to different due dates, whereas the job sequence determined by SPT depends only on the production time required by different types of elements, making the SPT sequence constant. The schedule plans provided by DQN and GA will also redefine the manufacturing order based on the specific demands of the orders. In contrast, the schedules produced by DQN and GA dynamically adjust the job sequence based on the specific requirements of each order. Unlike DPs, DQN and GA do not rely on a static sequencing policy. Instead, DQN uses a neural network-based policy, and GA applies iterative chromosome evolution to redefine the manufacturing order. This flexibility enables both DQN and GA to generate more optimal scheduling plans, tailored to the varying conditions of each task.

Compared to EDD and SPT methods, the optimal schedules generated by DQN or GA incur higher earliness penalties but achieve lower total penalties overall. This discrepancy suggests that to fully utilize the crew, completing certain jobs slightly ahead of schedule can reduce the overall total penalty. This conclusion is reflected in the training process of DQN, as shown in Figure 7–9. During the exploration phase of DQN training, in pursuit of higher rewards, the DQN agent continuously adjusts the balance between earliness and tardiness penalties. At the end of the training, the policy output by DQN tends to adopt the early completion of some PC elements. Although this approach may slightly increase the earliness penalty, it meanwhile reduces more tardiness penalty, thereby decreasing the total penalties. An illustrative example is shown in Figure 10. Assume there are 5 crew members, with two types of jobs requiring 2 and 3 crew members respectively, and different due dates. Figure 10a illustrates the job sequence according to EDD; Job 1 is completed before the due date, while Job 2 finishes later than the due date. Additionally, this schedule does not make full use of the crew resources. Figure 10b presents a more optimal schedule, where both Jobs are processed simultaneously, ensuring full crew utilization and completing both jobs before their respective due dates. This illustrative example explains why schedules generated by the DQN are superior to those created by DPs.

In the design of DRL, generalization ability has consistently been a major challenge. It is difficult to use a single DRL policy to address all production scheduling problems. In the context of precast facilities, however, incoming orders typically consist of a limited and standardized set of PC element types, with standardized dimensions as well. This paper focuses on scheduling and rescheduling precast orders under parallel production lines and limited resources conditions. The generalization ability of the proposed method is demonstrated by the fact that the pre-trained policy can be directly applied to new orders with varying amounts of precast elements and different due dates. However, as the initial conditions change, the pre-trained policy may not generate optimized schedules. To address this, the state design proposed in this paper incorporates completion rates and progress percentages to describe the production status of all precast elements, thereby mitigating the influence of the actual number of precast orders and their due dates in the state representation. This approach enhances the DRL algorithm’s ability to maintain high generalization when faced with different orders. The action design in the proposed method directly assigns jobs to production lines, which ties the action space to the number of production lines and the types of PC elements. For precast orders with the same type of precast elements, the action space remains fixed. Therefore, the proposed method can serve as a pre-trained policy that can be directly applied to precast orders that differ from the training tasks.

The proposed DRL-based scheduling method provides a practical and adaptable solution that can be directly implemented in precast factories with similar production structures. The implementation process involves two main stages: initial model training and policy deployment. During the training phase, production configurations—such as mold plate availability, labor crew settings, and PC order characteristics—are used to simulate the production environment. Once the model is trained, the pre-trained DQN agent can be deployed to handle ongoing and future orders without retraining. Production managers only need to input the new order data, and the system will automatically generate optimized scheduling plans through inference from the DQN policy.

In terms of system integration, the proposed DRL framework can be incorporated into existing factory management systems or production planning tools. Python-based interfaces and modular environment design allow for straightforward Application Programming Interface integration or standalone scheduling applications. With appropriate customization, factories using different mold configurations or product types can also retrain the model with their specific parameters using the methodology outlined in this study.

The practical implications of the proposed method extend to various construction participants:

1. Production managers can apply the pre-trained policy to instantly generate feasible and optimized production schedules. This reduces reliance on manual rule-based planning and enhances adaptability in the face of new or changing orders, improving decision-making efficiency in real-time factory operations.

2. Project managers and contractors benefit from increased reliability in the supply of PCs. The improved schedule consistency enables better coordination between off-site production and on-site assembly, helping to maintain construction timelines and reduce downtime.

3. Labor planners are supported by more balanced and transparent labor allocation. The model considers crew availability and task duration, which helps to avoid workforce bottlenecks and improve scheduling fairness, ultimately contributing to safer and more sustainable working conditions.

4. Policy makers and sustainability advocates may find the method useful for promoting efficient resource utilization and environmental responsibility. The reduction in idle time, early production, and unnecessary storage aligns with green building practices and sustainable construction regulations.

From the social and policy impact perspectives, optimizing precast production scheduling presents both opportunities and challenges that extend beyond operational efficiency. Socially, effective scheduling can enhance workforce stability by providing predictable labor demands, thereby improving job security and worker satisfaction. However, excessive optimization may inadvertently intensify workloads or reduce employment opportunities through automation, potentially undermining labor welfare. Furthermore, while streamlined production can minimize prolonged construction-related disruptions (e.g. noise and traffic congestion), compressed schedules risk extending work into antisocial hours, generating community grievances. From a policy standpoint, production scheduling must rigorously comply with labor regulations, occupational safety standards, and environmental directives. Algorithmic scheduling models must therefore embed these legal and ethical constraints to prevent violations. Notably, optimized scheduling aligns with broader sustainability objectives by reducing material waste and energy consumption, potentially qualifying projects for green building certifications or policy incentives.

While this study presents a novel and practical DRL-based approach for parallel precast production scheduling, several limitations must be acknowledged. These limitations pertain to four key areas: (1) algorithmic scalability and efficiency, particularly in handling large action spaces and bootstrapping issues inherent in DQN; (2) the narrow focus of optimization objectives, which currently considers only makespan and penalties for earliness and tardiness; (3) the simplified modeling of resource constraints, which excludes potential variability in material availability and process disruptions; and (4) the reliance on a fixed mold production mode, which may not generalize to more automated and efficient flow shop production environments. Recognizing these limitations provides valuable direction for future research aimed at enhancing the robustness, generalizability, and industrial applicability of DRL-based scheduling systems in the precast construction sector.

This study employs a single-agent DQN framework with an action masking mechanism to optimize production scheduling. However, several limitations exist concerning the model’s expressiveness and scalability. As a value-based method, DQN must evaluate all feasible actions, which becomes challenging as the scale of production increases. Although the proposed action space design effectively avoids the exponential growth commonly associated with complex combinatorial job assignments, it still maintains a strong dependency on the number of production lines and PC element types. In larger or more complex scenarios, the action space can still become significantly large. Even with action masking to filter invalid options, the agent must compute Q-values for a substantial number of valid actions at each decision step. This leads to longer training times and increases the risk of convergence to local optima. Additionally, traditional DQN suffers from bootstrapping flaws, which can lead to biases in Q-values computation, making the DQN training challenging to converge and decrease the rescheduling performance.

Since the limited existing research on DRL applications in parallel precast production scheduling optimization, this study focuses on two primary objectives: minimizing makespan and reducing total penalties associated with earliness and tardiness. Cases collected from an actual factory are utilized for performance validation. While these are critical metrics in practice, real-world applications often involve broader objectives. Factors such as energy consumption, carbon emissions, mold changeover costs, and labor overtime expenses are also essential for sustainable and cost-effective production. Future studies should incorporate multi-objective optimization to better reflect the complexity of real-world decision-making in precast factories.

This study models constraints related to the number of available mold plates and specialized labor crews. While these constraints represent key bottlenecks in most PC factories, other resource limitations can arise in practice. Although the surveyed factory possesses an independent concrete center and a reinforcement cage manufacturing center, ensuring resource supply in most situations, materials may still become scarce during peak demand periods. Additionally, the process times used in this study are treated as fixed and deterministic, which does not reflect real-world uncertainties caused by labor fatigue, equipment malfunctions, or supply delays. Consequently, future work should incorporate stochastic process times and additional resource constraints to create more robust and realistic scheduling models.

The cases for validation are based on fixed mold production mode, which remains widely used due to its lower site requirements and straightforward implementation. However, this method has the drawback of lower production efficiency, which is constrained by the number of mold plates and skilled crews. Field observations revealed that some factories are transitioning toward mechanized flow shop production, which leverages automation to improve throughput and resource utilization. Therefore, future work should extend the current DRL framework to address the unique scheduling complexities and optimization opportunities in parallel flow shop production environments.

To bridge existing gaps in the field, future research could focus on integrating multi-agent deep reinforcement learning and developing more sophisticated DRL frameworks to enhance the training process and address complex challenges. The application of multi-agent DRL shows promise in reducing the complexity of high-dimensional action spaces, thus simplifying neural network architectures and streamlining the DRL training. Enhanced frameworks, such as Double DQN, Noisy Networks, and Prioritized Experience Replay, could be employed to tackle bootstrapping problems in intricate scenarios. Additionally, refining current algorithms through simulation-based GA could tailor solutions for parallel production tasks, while the advancement of multi-objective optimization algorithms leveraging DRL offers a pathway to addressing more nuanced objectives. There exists a substantial potential to enhance labor efficiency and address uncertainties, thereby rendering optimization strategies increasingly relevant and adaptable within actual production environments. Hence, it is crucial for forthcoming studies to explore not merely parallel fixed mold production but to also broaden their investigation to encompass parallel flow shop production, thereby enriching the research domain with multifaceted optimization methodologies. To address the improvement of pre-trained DQN policies in unforeseen scenarios, several effective strategies could be incorporated, including domain randomization to expose the policy to varied scenarios during training; data augmentation methods to enrich the training dataset with diverse state-action experiences; robust adversarial training to explicitly handle challenging or novel states; regularization techniques (e.g. dropout, entropy regularization) to mitigate overfitting; and meta-learning or transfer learning approaches that enhance policy generalization by training across multiple related environments. Employing these approaches can improve the DQN policy’s adaptability and robustness in unforeseen cases.

This study proposed an automatic parallel precast production scheduling method using DQN to minimize production makespan and penalties of earliness and tardiness. The proposed DQN framework considered various constraints encountered in real-world factories, including resource constraints, labor constraints, and working hour limitations. A simulation environment was developed to replicate the precast production process under these constraints. Building on this, an improved simulation-based GA was presented for comparison with the DQN method. The proposed DQN method is designed to obtain a pre-trained policy that can directly generate optimal production schedules with minimized earliness and tardiness penalties as well as shortened makespan. The stability and feasibility of the DQN method were validated by empirical experiments conducted in actual factories, including several rescheduling tasks. When compared with GA and traditional DPs like EDD and SPT, the DQN method outperformed in terms of effectiveness and efficiency in generating schedules. The proposed DRL framework also exhibits rescheduling capabilities that can accommodate diverse and dynamic order requirements. Real-world case data collected from a surveyed factory were used to further validate the practical applicability of the proposed approach.

Future research could explore the use of innovative DRL frameworks to improve production scheduling. Research on parallel flow production is another direction that merits attention. As the Internet of Things continues to drive automation in precast production, the application of DRL methods is expected to have a transformative impact on the industry’s future development. Additionally future work should consider diversifying scheduling objectives, such as incorporating carbon emissions into the optimization framework. Finally, integrating production scheduling with construction scheduling could lead to the development of a unified scheduling system, supporting optimal decision-making across the entire industrial supply chain.

The authors wish to acknowledge the financial support from the Australian Research Council (ARC), Australian Government (Nos: DP200100057, IH200100010; FT220100017; IC240100020; DP250101775).

Source(s): Authors’ own work

For better illustrate the state transition, here presents an example of how State $S$ is determined in real simulation. Let’s assume a precast order with 2 types of elements sent to 3 parallel production lines. The total number of each type is 5 and 10 respectively; and due dates is 72 h and 120 h respectively. At the beginning moment $t=0$⁠, there are no PC being processed; and thus the $S0$ is:

Each roll in $S$ represents a single production line, the −1 values represent that currently no job is assigned to any production line. Then the agent choose action $a0=[1,2]$⁠, which means 1 and 2 PC elements of two types are assigned respectively. Assuming the first process times (mold assembling) of the PCs are 2 and 3 respectively, then after action execution, the $S1$ is:

The first column of $S$ represents the PC codes. The second column represents the current completion/processing ratio of each type of PC. The third column represents the relative value between the current time and due dates. The last six columns represent the completion rates of each production process.

Source(s): Authors’ own work