Federated Learning Framework for Privacy-Preserving Threat Detection in Distributed IT Systems

1. INTRODUCTION

Cloud The IT infrastructure has experienced a radical shift, which has become highly distributed due to cloud platforms, edge devices, and geographically distributed enterprise networks [1, 2]. Organizations are progressively using hybrid and multi-cloud systems, remote access, and interconnected digital ecosystems to facilitate scalability, stability, and operational effectiveness [3, 4]. Although this architectural redesign allows flexibility and performance, it also increases the area of the cyber-attack, and distributed IT systems become one of the main targets of more advanced threats.

In order to address these threats, intrusion detection and threat forecasting systems based on machine learning (ML) have become a key element of modern cybersecurity policies. By learning patterns from historical network traffic and system logs, ML models detect abnormal and malicious behaviour more accurately than rule-based systems [5, 6]. Most of the current ML-based systems of intrusion detection (IDS) are, however, based on centralized learning models whereby raw security data is gathered into one repository.

The centralized collection of data is now being limited more by privacy laws, corporate policies, and trust limits [7]. There are also laws like the General Data Protection Regulation (GDPR) and industry-related compliance laws that limit the distribution of sensitive network logs, user behavioural traces, and operational metadata [8]. Even in the absence of regulatory barriers, organizations are often reluctant to disclose raw security data due to intellectual property concerns, reputational risk, and potential misuse. Because of this, security data is often isolated by departments, subsidiaries, or partner organizations, which reduces the usefulness of centralized analytics.

These limitations have inspired an increasing amount of interest in privacy-preserving learning paradigms, especially federated learning (FL). In federated learning, several parties jointly train a global model, but retain raw data on a local scale at each location. Participants, instead of sharing data, share model parameters or updates, which diminishes the chances of directly exposing data. This paradigm is particularly attractive to distributed IT systems, where the cooperation across the boundaries between organizations can enhance threat awareness without breaking privacy or governance restrictions. However, the federated learning implementation for real-world and practical threat detection in distributed settings is still a challenge that is open and poorly investigated.

According to [9], machine learning has proved to be effective in cybersecurity, the existing intrusion detection systems have inherent challenges when used in distributed IT systems. The centralized IDS designs implicitly presuppose the possibility of collecting, storing, and processing security data from all sources in one location. This is becoming an unrealistic assumption in the context of regulatory constraints, decentralized ownership and heterogeneous infrastructure.

Some serious threats are posed by centralized learning strategies. To start with, raw traffic information and logs can be directly aggregated, which can breach the privacy rules or the internal security policies [10]. Second, repositories that are centralized are valuable assets, and the damage of data breaches becomes even greater [11]. Third, centralized systems demand high-bandwidth data transfer and continuous synchronisation, which can be impractical or expensive in large-scale distributed deployment [12].

In the other extreme, local-only learning, in which each organization or site trains its own isolated model, also has its fair share of downsides. The local models have the disadvantage of low visibility, since they are trained on small and possibly biased data sets [13]. Consequently, they tend to miss new or unusual attack patterns, which can only be discovered when the information is consolidated across a number of sites. Such fragmentation results in inconsistency in the performance of threat detection and compromises the system-wide security position.

To add to these issues, security data in organizations is non-independent and identically distributed (non-IID). Various locations have dissimilar traffic patterns, workloads, and attack profiles, as determined by their roles, configurations and exposure to threats. This heterogeneity makes collaborative learning difficult, and model convergence and performance are, in most cases, poor when not specifically taken care of. This creates a need for learning frameworks that operate effectively under non-IID conditions while respecting privacy and trust constraints in distributed IT systems.

There is a lot of literature on machine learning-based intrusion detection, but to a large extent, it is accuracy-focused and presupposes centralized data access. Most of the suggested IDS models are only tested using predictive performance measures without much thought of implementation, privacy or cost of the system. Moreover, much of the previous literature is based on excessively simplistic data that is not representative of modern network traffic.

More current research has investigated federated learning for use in cybersecurity, such as intrusion detection. Although these attempts prove the theoretical viability of FL-based IDS, there are still a number of limitations. A lot of FL-IDS research is theoretical or based on experimental systems on a small scale. Assessments do not take into account realistic non-IID data distributions, communication overhead, and convergence behaviour, which are factors that strongly affect practical implementation. Moreover, few studies have proven federated IDS solutions on recent and extensive data like UNSW-NB15, which reveals various types of attacks and the actual traffic features. Consequently, no end-to-end empirically validated federated learning architectures that collectively consider the accuracy of threat detection, privacy protection, system-level effectiveness, and data heterogeneity on distributed IT settings have been proposed. Such a gap spurs the current research.

To fill the given gaps, the following key contributions were made in this paper:

1. System-level federated IDS evaluation, analysing convergence, communication overhead, and round efficiency.
2. Empirical analysis of non-IID impact, demonstrating stable convergence under heterogeneous client distributions.
3. Communication-efficient federated IDS evaluation, explicitly quantifying communication overhead and convergence rounds under realistic non-IID client distributions.
4. Reproducible federated intrusion detection pipeline using UNSW-NB15 under realistic deployment constraints.

These contributions collectively distinguish this work from prior FL-IDS studies, which typically focus on detection accuracy alone without jointly addressing system-level efficiency, non-IID convergence behaviour, communication overhead, and deployment feasibility within a single reproducible evaluation. While FedAvg and DNN architectures are individually well-established, their integration within an end-to-end empirically validated federated pipeline under realistic heterogeneous conditions with explicit quantification of communication and computation cost represents the primary novelty of this study.

The paper mainly explores the possibility of using federated learning to facilitate an effective and privacy-conscious threat detection in distributed IT systems. In particular, the research questions that the study attempted to answer included:

RQ1: Can federated deep neural networks achieve competitive threat detection performance compared to centralized learning approaches?

RQ2: How does non-IID data distribution across distributed clients affect federated learning convergence and accuracy?

RQ3: What is the system-level costs, in terms of communication and computation, associated with federated intrusion detection?

RQ4: What privacy–utility trade-offs arise when applying federated learning to threat detection tasks?

2. LITERATURE REVIEW

2.1. Machine Learning and Deep Learning for Intrusion Detection

The IDS has largely developed out of signature-based and rule-driven systems to data-driven systems that can recognize sophisticated and hitherto unseen attack patterns [14]. Conventional IDS methods, although useful in the background of known threats, are incapable of adapting to the dynamic threat environment and the growing size of contemporary network traffic [15, 16]. This shortcoming has induced the use of machine learning (ML) methods, which acquire discriminative patterns directly from data and have the ability to extrapolate beyond prescribed rules.

The initial ML-based IDS solutions were based on classical algorithms, including decision trees, support vector machines, and ensembles [15]. The techniques proved to be more accurate in detection than rule-based systems, especially in the detection of anomalies. Nevertheless, they are typically limited in their ability to represent complex and high-dimensional relationships in network traffic data due to the need to feature engineer and capacity limitations. These constraints were emphasized as the network environments became more heterogeneous and dynamic in terms of traffic patterns.

The most recent developments in deep learning have also changed the field of IDS research. Convolutional architectures, deep neural networks (DNNs) and recurrent models have been demonstrated to learn non-linear dependencies and time-dependent correlations of network flows [17]. DNN-based IDS models are especially suitable for tabular network traffic datasets, where interactions between multiple features are the drivers of attack behaviour, as opposed to individual indicators [18, 19]. Empirical studies consistently show that deep learning outperforms traditional machine learning methods.

Although these have been made, most deep learning-based IDS studies have centralized assumptions regarding access to training data. The models are usually trained on aggregated traffic logs, which are obtained through several sources, and an assumption that is becoming unrealistic in distributed IT systems [19, 20]. As a result, deep learning helps to increase the detection power, but it does not help to resolve the underlying issues of privacy, trust, and data governance that occur in practice.

2.2. Centralized Learning Limitations in Distributed IT Environments

The centralized learning architectures are still prevalent in the IDS literature because of their conceptual ease and excellent empirical results. Centralized models have the advantage of having an all-encompassing view of various attack patterns by consolidating the data from a variety of sources [21]. Nonetheless, there are major practical and security-related issues presented by this paradigm when it is used in relation to a distributed IT system [22].

First, the principle of data aggregation in a centralized manner is in direct contradiction with the privacy laws and the internal governance policies [10]. Network traffic logs can consist of sensitive user, application or organizational process information. Moving such data to a central point enhances the chances of data leakage and non-compliance with regulations. Second, centralized repositories generate desirable targets to the adversaries, increasing the magnitude of the consequences of a successful intrusion [11]. Third, the ongoing exchange of high-volume network traffic data is very expensive in terms of bandwidth and infrastructure expenses [12].

Centralized approaches also conceal the heterogeneity of distributed environments as seen in a modelling view. Non-IID characteristics in security data are frequently observed when data is collected by different organizations or network segments because of the difference in the workloads, configurations and vulnerability to threats. Aggregation [23]. This can cause the loss of local site-specific patterns, which would diminish the applications of local threat detection. The limitations highlight the necessity of alternative learning paradigms, which would be effective to work without central data gathering.

2.3. Federated Learning: Principles and Security Applications

Federated learning has become a potential paradigm of collaborative model training under privacy conditions. In FL, several clients use local models to train local data and periodically provide updates on those models to a central server, which combines the updates to create a global model [24]. Raw data never leave the client environment, and sensitive information is not directly exposed.

Federated learning was initially suggested in mobile and edge computing situations, but it has since been researched in many other areas, such as healthcare, finance, and cybersecurity. FL provides a promising mechanism for sharing threat intelligence across organizational boundaries and maintaining the locality of data in the context of security analytics. FL allows collective learning without the need to explicitly share data by aggregating learned representations instead of raw logs.

Nevertheless, federated learning comes with its own set of issues. The non-IID data distributions among clients may slow the convergence and deteriorate the performance of the model [25]. It can add a lot of overhead to communication between clients and the server, especially with deep learning models. Also, FL decreases the direct exposure of the data; however, the information leakage occurs when the model updates itself [26]. These issues require stringent empirical testing in the use of FL to security sensitive applications like intrusion detection.

2.4. Federated Learning for Intrusion Detection Systems

An emerging body of work has investigated the application of federated learning to IDS. The current literature on FL-IDS typically shows that federated models can be superior to local-only models and can achieve the performance of centralized systems in some circumstances [27]. These results confirm the effectiveness of FL as a privacy-saving substitute for cooperative intrusion detection.

However, there are still a number of weaknesses in the existing literature. Numerous studies test FL-IDS techniques in simplified experimental conditions, e.g., balanced data partitions or a few simulated clients [28, 29]. Practically, security data in organizations is very heterogeneous and realistic non-IID conditions are not often discussed in detail [29]. Besides, performance appraisal commonly pays too much attention to accuracy-based measures, but very little consideration to convergence behaviour, communication cost, or computational overhead.

A second significant limitation is the selection of the dataset. Much of the FL-IDS literature is based on old standards that are no longer relevant to modern attack vectors and network actions. The applicability to real-world conditions is not as high, since modern datasets, with different types of attack and realistic traffic properties, are used rather sporadically [30].

2.5. UNSW-NB15 Dataset in IDS Research

The UNSW-NB15 dataset is an important improvement of the previous intrusion detection benchmarks. It captures contemporary network traffic created under controlled conditions, which are realistic, but it contains a wide range of categories of attack, as well as normal behaviour [31]. The dataset is rich in features (including flow-based, protocol-level, and content-related) and is therefore appropriate to test the performance of sophisticated ML and deep learning models.

Previous IDS research on UNSW-NB15 has shown that it is viable in training and assessing the intrusion detection models [32]. Nevertheless, the majority of the available literature uses centralized learning paradigms and thus shares the privacy and scalability drawbacks mentioned above. The existing body of work that has covered federated learning with UNSW-NB15 is very limited, with those studies that do not typically analyse the system-level or assume realistic heterogeneous clients.

This is a major gap, especially considering that the dataset is suitable for modelling the distributed environment where various clients can be experiencing varying subsets of the traffic and types of attacks. The application of UNSW-NB15 in a federated environment offers a chance to test the performance of IDS in an environment that is close to real-life distributed IT systems.

2.6. Research Positioning

Although machine learning and deep learning have significantly advanced the intrusion detection processes, the current IDS research is full of centralized assumptions that cannot be used in contemporary distributed IT settings. Federated learning provides a more promising solution since it allows model training together without the exchange of raw data. Nevertheless, the existing FL-IDS literature is constrained by simplified assumptions, old-fashioned datasets, and a small number of evaluation criteria.

This paper addresses these weaknesses by introducing an end-to-end federated learning system to detect malicious traffic on the UNSW-NB15 dataset in realistic non-IID scenarios. With the addition of system-level metrics, privacy considerations, and predictive performance, the proposed solution advances the state of the art in practical and privacy-aware intrusion detection within distributed IT systems. Several important related research threads have emerged in the recent literature that contextualise and motivate the design choices made in this work.

From a privacy-preservation perspective, several recent studies have explored stronger formal mechanisms within FL-IDS frameworks. Differential privacy (DP) has been integrated into federated intrusion detection pipelines through calibrated noise injection and privacy budget accounting, providing measurable guarantees against gradient-based inference attacks [27, 29]. Secure aggregation protocols have been applied to prevent the central server from observing individual client updates, offering an additional layer of confidentiality. In contrast, the present study deliberately omits these mechanisms to serve as a controlled baseline that isolates the effects of federated averaging and non-IID data distribution on system performance, without the confounding influence of privacy noise on model convergence.

Regarding non-IID heterogeneity, recent work has proposed alternative aggregation strategies that improve convergence under heterogeneous client data. FedProx introduces a proximal regularisation term that penalises local models for deviating too far from the global model, improving stability under heterogeneity. SCAFFOLD uses variance reduction through control variates to correct for client drift caused by non-IID distributions. These methods demonstrate measurable improvements over standard FedAvg in heterogeneous settings. The present study adopts FedAvg as a deliberate baseline choice to enable clean and interpretable system-level analysis; future work should benchmark the proposed pipeline against FedProx and SCAFFOLD to quantify any performance and convergence gains under the non-IID conditions used here.

3. METHODOLOGY

3.1. Framework Overview

The proposed methodology takes the federated learning paradigm to allow privacy-preserving threat detection over distributed IT systems. The objective is to jointly train a global intrusion detection model without exchanging raw network traffic among participants. Local model training by each distributed client, which is a separate organizational node or network segment, is done on its own data, and a central coordinating server combines the learned parameters to update a shared global model (Fig. 1).