The performance comparison between behavior cloning trained with asymmetric loss and focal loss with different hyperparameter settings.
| Dialogue Policy Objective | Hyperparameter Setting | Success Rate (%) | Inform | Complete Rate (%) | Booking Rate (%) | Average Turn (Succ/All) | ||
|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | ||||||
| ω+ =0, ω− = 2, m = 0.05 | 83.0 | 82.4 | 93.0 | 85.1 | 91.4 | 91.1 | 11.4/12.2 | |
| ω+ =0, ω− = 4, m = 0.05 | 82.0 | 75.3 | 90.8 | 79.4 | 87.7 | 89.1 | 11.1/12.0 | |
| Asymmetric loss | ω+ = 0, ω− = 5, m = 0.05 | 82.5 | 79.2 | 93.1 | 83.1 | 90.8 | 90.7 | 11.4/12.3 |
| ω+ = 1, ω− =2, m = 0.05 | 82.1 | 80.4 | 92.8 | 83.8 | 90.4 | 90.1 | 11.3/12.6 | |
| ω+ =1, ω− =4, m = 0.05 | 82.0 | 73.1 | 92.4 | 78.9 | 90.0 | 89.9 | 11.3/12.0 | |
| ω+ = 1, ω− = 5, m = 0.05 | 80.2 | 64.6 | 90.2 | 72.2 | 86.7 | 90.0 | 11.5/12.0 | |
| Focal loss | ω+ = 2, ω− = 2, m = 0 | 83.3 | 86.7 | 93.4 | 88.6 | 92.0 | 91.8 | 11.4/12.3 |
| Dialogue Policy Objective | Hyperparameter Setting | Success Rate (%) | Inform | Complete Rate (%) | Booking Rate (%) | Average Turn (Succ/All) | ||
|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | ||||||
| 83.0 | 82.4 | 93.0 | 85.1 | 91.4 | 91.1 | 11.4/12.2 | ||
| 82.0 | 75.3 | 90.8 | 79.4 | 87.7 | 89.1 | 11.1/12.0 | ||
| Asymmetric loss | 82.5 | 79.2 | 93.1 | 83.1 | 90.8 | 90.7 | 11.4/12.3 | |
| 82.1 | 80.4 | 92.8 | 83.8 | 90.4 | 90.1 | 11.3/12.6 | ||
| 82.0 | 73.1 | 92.4 | 78.9 | 90.0 | 89.9 | |||
| 80.2 | 64.6 | 90.2 | 72.2 | 86.7 | 90.0 | 11.5/12.0 | ||
| Focal loss | 11.4/12.3 | |||||||
Sharing content requires targeting cookies to be enabled. Please update your cookie preferences to use this feature.