Evaluation and Calibration of Cross-Dataset Robustness in IoT Intrusion Detection Systems: A Deployment-Aware Approach

This study presents a hypothesis of granularity asymmetry: NF-ToN-IoT-v2 flows are exporter-specified summaries of steady and aggregated statistics across full connections. Conversely, Edge-IIoTset pseudo-flows are heuristic windowed approximations built on packet-level and protocol-level logs. Although the nominal similarity of the names of features in both representations may be similar (e.g., the number of packets, count of bytes proxies, and duration), the statistical meaning of the term is significantly different. Consequently, NF flow representations are not transferred to the pseudo-flow distributions of Edge telemetry.

Qualitative evidence for this interpretation is provided by the cross-dataset behaviour in the common canonical feature space. The severe deterioration in the ranking performance (NF-AUC of approximately 0.45) in the NF-Edge direction, as opposed to the virtually perfect performance in the source domain, shows that there is a substantial distributional mismatch between NF flows and Edge pseudo-flows. This discrepancy indicates that the marginal distributions and semantic meanings of key flow statistics vary across datasets, although feature names may have exact matches. In turn, unmonitored threshold adaptation proves to be able to correct decision-level calibration, but is unable to restore ranking quality in case the representation semantics are not aligned.

It must be mentioned that positive-rate matching implicitly assumes that the target deployment environment has a more or less widely similar prevalence of attacks or alert budget as measured during source validation. When the actual attack rate in the target domain varies significantly, the adjusted threshold will cease to correspond to an optimal operating point. Therefore, positive-rate matching must not be considered a solution to the distribution shift but rather a pragmatic solution for matching calibration.

These results lead to a crucial conclusion: when the semantics of features vary, cross-dataset intrusion detection transfer is not symmetrical, even when feature names are similar. To overcome this weakness, more principled feature harmonisation or domain adaptation schemes are necessary which explicitly consider variations in the granularity of telemetry and involve not only the use of common feature schemas.

5.11. Per-Attack Precision/Recall Analysis

A per-attack analysis was conducted on the NF-ToN-IoT-v2 source-IP GroupSplit test set. The attack family field was not used as an input feature during training because it would create label-derived leakage. Instead, the attack family labels were retained only for post-hoc diagnostic evaluation after the binary benign/attack predictions were generated. Therefore, this analysis shows how well the binary IDS decisions cover different attack categories under the operational GroupSplit setting.

The strongest per-attack performance was observed for DDoS and Botnet traffic. This is expected because high-volume attack behaviour often produces clear deviations in the packet counts, byte statistics, TCP flag behaviour, and timing-related flow features. Reconnaissance achieved comparatively lower recall, suggesting that low-intensity scanning and probing activities are more difficult to distinguish from benign, background communication. These results support the need to report per-attack diagnostics in addition to aggregate binary metrics, especially when the dataset contains attack families with different traffic intensities and behavioural signatures (Table 4).

Table 4. Per-attack precision, recall, and F1-score.

5.12. Comparison with Baseline Models

To evaluate the relative performance of the DAE-DFE, several conventional and recent deep-learning IDS baselines were compared under the same source-IP GroupSplit protocol. All models used the same preprocessing pipeline, feature configuration, training, validation, and test splits to ensure a fair comparison. The baseline set included classical machine learning models, a multilayer perceptron, and recent deep-learning IDS architectures, including CNN, LSTM, Transformer, and autoencoder-based IDS models.

The comparison shows that DAE-DFE is competitive with recent deep-learning IDS baselines under the source-IP GroupSplit protocol. However, it is important to avoid overstating these results. DAE-DFE does not achieve the highest raw accuracy because the MLP and Transformer baselines produce comparable or slightly higher accuracy values. Its strongest performance appears in recall, F1-score, ROC-AUC, and PR-AUC, where it remains one of the best-performing models. Therefore, the advantages of the proposed framework should not be described as simple superiority over every baseline. Instead, its main value lies in the combination of strong detection performance with deployment-aware evaluation, host-split testing, cross-dataset transfer analysis, calibration curves, ablation analysis, and unsupervised threshold adaptation (Table 5). DAE-DFE performs competitively against the baseline models and achieves the strongest or near-strongest results for recall, F1-score, ROC-AUC, and PR-AUC, under the GroupSplit protocol. However, the raw accuracy of the DAE-DFE is slightly lower than that of the MLP baseline and comparable to that of the transformer baseline. Therefore, the results should be interpreted as evidence of strong and stable detection performance, rather than absolute dominance across every metric. The main contribution remains the deployment-aware evaluation and calibration methodology, rather than only marginal metric improvement over deep learning baselines.

Table 5. Baseline comparison under the source-IP GroupSplit evaluation protocol.

5.13. Statistical Validation Across Multiple Runs

To address the possibility that the reported results may depend on a single favourable random split or model initialisation, the main experiments were repeated using five independent random seeds. For each run, the same preprocessing pipeline, feature configuration, model architecture, validation-based threshold selection, and evaluation protocol were applied. The results are reported as mean ± standard deviation for the principal threshold-dependent and threshold-independent metrics (Table 6).

Table 6. Statistical stability of DAE-DFE across repeated runs.

The repeated-run results show that the NF-ToN-IoT-v2 experiments were highly stable across random seeds. The Operational IID, Generalisable IID, and source-IP GroupSplit produced very low standard deviations, indicating that the within-dataset findings were not dependent on a favourable initialisation or a single split. This supports the robustness of the reported NF-ToN-IoT-v2 results, although the near-saturated performance should still be interpreted considering the strong separability of this benchmark.

Greater variability was observed in the cross-dataset settings, especially in the NF→Edge transfer. This is expected because cross-domain deployment is more sensitive to score calibration, source-target feature mismatch, and pseudo-flow approximation errors. In the Edge→NF direction, direct threshold transfer remained unstable, but unsupervised threshold adaptation consistently improved the decision-level performance while leaving the ROC-AUC and PR-AUC unchanged. This confirms that the Edge→NF failure under direct transfer was mainly caused by calibration drift rather than a complete ranking failure. In contrast, NF→Edge remained weak even after adaptation, indicating that threshold correction cannot compensate for the deeper feature-semantic mismatch between exporter-defined NetFlow records and packet-derived pseudo-flows.

6. DISCUSSION

6.1. Key Findings

The robust performance of DAE-DFE on NF-ToN-IoT-v2 can be explained by its capability to learn nonlinear representations of interacting flow-level statistics that cannot be effectively represented by linear decision boundaries. Traffic features, such as packet counts, byte volumes, duration, and TCP flag patterns, are strongly coupled and heavy-tailed; therefore, the compact latent representation provides a lower-dimensional space in which nonlinear traffic interactions can be represented more effectively than in the original feature space. The powerful preprocessing pipeline, which consists of quantile clipping, median imputation, and RobustScaler normalisation, strengthens this effect by stabilising the optimisation in the face of the extreme value distributions characteristic of NetFlow telemetry.

Meanwhile, the near-saturated F1 and ROC-AUC numbers should be explained considering the established characteristics of NF-ToN-IoT-v2. The dataset contains a large proportion of attack traffic and exhibits strong separability between benign and malicious flows, and multiple classifiers, not just deep models, exhibit very high performance. The minimal degradation at the operational group split (SrcIP) protocol thus alludes to the fact that discriminative signals are mostly global as opposed to host-specific, and not to residual leakage or memorisation. This is evident in comparison to the evaluation based on the same GroupSplit environment, where the classical models also score comparably high.

In addition to in-dataset performance, cross-domain deployment demonstrates another clearly different and practically important challenge: calibration shift. Within the Edge→NF transfer configuration, the model still maintained a significant ranking capability (ROC-AUC = 0.80; PR-AUC = 0.80), but direct threshold transfer led to near-random accuracy of decisions because of an aggressive positive bias in the score distribution. This observation is consistent with the larger calibration results in machine learning, and under distribution shift, thresholds that are optimised in one domain are not optimised. The suggested unsupervised threshold adaptation method, which relies on positive-rate matching on the unlabelled target scores, reduces this problem without retraining or having labelled target scores and is therefore applicable to a realistic IoT implementation.

Finally, it can be stated that generalisation is limited by the difference in feature granularity and semantics based on the observed asymmetry in cross-dataset transfer (NF to Edge weak, Edge to NF moderate). NetFlow summaries represent exporter-specified aggregation, and edge pseudo-flows are aggregated by heuristic windowing of packet/protocol logs, adding noise to approximations, and creating features that are not matched. This asymmetry motivates the development of more principled methods of feature harmonisation and domain adaptation to enhance the robustness of heterogeneous sources of IoT telemetry. This asymmetric behaviour of transfer highlights the fact that telemetry granularity and feature semantics are the main limitations of cross-dataset generalisation in IoT IDS and not model capacity.

6.2. Ablation Study

To investigate the effects of regularisation in the suggested framework, an ablation study was implemented by selectively eliminating the two auxiliary elements of the DAE-DFE: denoising through noise injection when using Gaussians and reconstruction-based regularisation. Four versions were tested: the complete DAE-DFE model, the model without reconstruction loss, the model without denoising, and the pure classifier with neither of the regularizers. They were evaluated using the strong NF-ToN-IoT-v2 GroupSplit (SrcIP) protocol and cross-dataset Edge→NF transfer.

In all settings, the variants achieved almost the same peak performance, with F1 scores clustered around 0.992 on NF-ToN-IoT-v2 and around 0.898 on unsupervised threshold adaptation cross-dataset deployment (Table 7). Across the ablation variants, the absolute differences in the F1-score, ROC-AUC, and PR-AUC were very small, generally below 0.001 in the reported headline metrics. As no formal hypothesis test was conducted, these differences should not be described as statistically significant or insignificant. Instead, the results indicate that removing denoising, reconstruction regularisation, or both, produced only negligible practical changes in the reported metrics. This suggests that the main discriminative performance is primarily driven by the supervised latent representation and the strong separability of the flow-statistic features, whereas denoising and reconstruction act mainly as stability-oriented regularisation components.

Table 7. Ablation results (DAE-DFE). Ablation of DAE-DFE training components. “Full” includes denoising + reconstruction. “No Reconstruction” sets λ_rec=0. “No Denoising” sets σ=0. The Pure Classifier” removes both.

Note: Ablation experiments were conducted using a different random seed; therefore, slight metric variations were expected.

The ablation results show that removing reconstruction regularisation, denoising, or both, causes only marginal changes in the headline performance. This indicates that the main discriminative performance is driven primarily by the learned supervised latent representation and separability of the evaluated flow-level features. Therefore, denoising and reconstruction regularisation should be interpreted as stability-oriented training components, rather than the sole source of performance improvement. This finding also supports the revised novelty framing of this paper: the main contribution is not the invention of a new architecture but the deployment-aware evaluation and calibration methodology.

6.3. Computational Complexity and Deployment Feasibility

Practical IDS deployment requires not only high detection performance but also feasible training and inference costs. The DAE-DFE introduces additional training overhead compared with a pure classifier because the model optimises both the classification and reconstruction objectives. However, during inference, a decoder is not required for probability-based intrusion detection unless reconstruction monitoring is explicitly used. Therefore, the operational inference path consists only of the preprocessing pipeline, encoder, and classifier head.

The encoder–decoder structure introduces an additional training cost compared with a pure classifier because the model jointly optimises the classification and reconstruction objectives. However, during deployment, a decoder is not required for probability-based intrusion detection unless reconstruction monitoring is also enabled. Therefore, the operational inference path consists only of preprocessing, an encoder, and a classifier head. This makes the model suitable for edge-assisted or gateway-level IDS deployment, where low inference latency is more important than offline training time (Table 8).

Table 8. Computational profile of the DAE-DFE framework.

6.4. Limitations and Threats to Validity

Several limitations and threats to validity should be considered when interpreting these results. First, the balanced sampling strategy was useful for controlled comparison and stable model training; however, real IoT networks are usually highly imbalanced, with benign traffic substantially outnumbering the attack traffic. Therefore, deployment thresholds may require further tuning to achieve realistic prevalence levels and operational alert budgets. Second, NF-ToN-IoT-v2 appears highly separable at the flow-statistic level, which explains why very high F1-score and ROC-AUC values can be obtained even under stronger evaluation protocols. Therefore, these results should be interpreted as strong performance on the evaluated benchmark rather than universal evidence of real-world IDS effectiveness. Third, Edge-IIoTset pseudo-flow generation provides a practical way to conduct cross-dataset evaluation; however, it remains an approximation of NetFlow-style telemetry and may not fully preserve packet-level timing or connection semantics. Fourth, positive rate threshold adaptation assumes that the expected alert rate or attack prevalence in the target domain is reasonably close to the source validation setting. If the true target domain attack prevalence changes substantially, the adapted threshold may become too conservative or aggressive [16].