Federated Optimization for On-Device Edge Industrial Network: A decentralize Federated Learning Approach

Introduction:

Federated learning is a machine learning technique that allows multiple devices or systems to collaboratively train a model without sharing their data. This is particularly useful in the context of advanced manufacturing, where many devices and systems may be connected through the Internet of Things (IoT) and generate large amounts of data.

Federated Learning can be implemented using two different architectures, Centralized Federated Learning and Decentralized Federated Learning.

In centralized federated learning, data from multiple devices or servers is sent to a central location for processing and training. This central location could be a single server, a cluster of servers, or a cloud-based service. The central server is responsible for aggregating the model updates from the devices, computing the new model parameters, and then sending the updated model back to the devices. This approach is often used when the data is highly sensitive or the devices have limited computational resources.

                                                                                            Figure 1: Centralize Federated Learning Architecture

An example of centralized federated learning can be found in work by Google on improving the performance of its voice recognition system. The company used a centralized approach to train a neural network on voice data from multiple devices in this case. The data was collected by the devices, sent to a central server, and combined to improve the accuracy of the voice recognition system.

In decentralized federated learning, the data remains on the devices themselves, and the model is trained locally on each device using the data available on that device. The local models are then combined to form a global model without needing a central server. This approach is often used when there are privacy concerns or limitations on bandwidth or communication.

To use federated learning for advanced manufacturing based on industrial IoT data analytics, it is necessary to have a distributed system of devices or systems connected through the IoT and generate data that can be used to train a machine learning model. The data may include sensor readings, production metrics, maintenance logs, and other data relevant to the manufacturing process.

                                                                                                     Figure 2: decentralize Federated Learning Architecture

On the other hand, an example of decentralized federated learning can be found in work by Apple on improving its intelligent keyboard feature. In this case, the company used a decentralized approach to train a model on user input data from multiple devices. The data was processed locally on the devices themselves, with each device contributing to the overall model.

Background:

Predictive maintenance is a critical component of industrial settings that can lead to significant cost savings and improved operational efficiency. According to a study by McKinsey & Company, predictive maintenance can reduce maintenance costs by up to 30%, reduce unplanned downtime by up to 50%, and increase equipment availability by up to 10-20% (Parida et al., 2019).

Maintenance can be scheduled proactively by predicting when equipment will likely fail, reducing the need for unscheduled repairs and minimizing unplanned downtime. This can help increase the reliability and efficiency of industrial equipment while improving operational safety.

Predictive maintenance is an important technique that allows industrial companies to track the performance of machine components in real time and minimize repair downtime. According to a study by the International Journal of Engineering Research and General Science, predictive maintenance can increase equipment availability by up to 20% and reduce maintenance costs by up to 25% (Mittal et al., 2016).

By analyzing data from sensors and other sources, predictive maintenance tools can help enterprise owners determine the working conditions of equipment and machines, allowing them to maximize uptime and increase efficiency. These tools can also prevent costly operational interruptions and equipment failures by providing maintenance arrival alerts, allowing for proactive maintenance planning and reducing the need for unscheduled repairs.

According to www.precedenceresearch.com , the predictive maintenance market size will rise approximately to 62.21 Billion by 2030.

                                                                                             Figure 3: Predictive Maintenance Market Size

Why are traditional approaches like centralized machine learning not well-suited to these environments? 

Centralized setting: In a centralized setting, a central server stores all the data and performs all the computations required for training a model. This means that all the data must be transmitted to the central server, which can be a major bottleneck in scenarios where data is generated at the network's edge or with limited communication resources. Additionally, the centralized server may be a point of failure or a security risk, as all the data is stored in a single location.

Some challenges in the centralized are:

Large amounts of data: Industrial equipment can generate large amounts of sensor data, and transmitting all of this data to a centralized server can be impractical or infeasible. This can lead to high communication bandwidth requirements, which may not be feasible in low-bandwidth or high-latency environments.

Real-time processing: In many industrial settings, decisions must be made in real-time based on the sensor data, which requires fast processing times. Centralized machine learning approaches may not be able to provide the real-time processing required for some applications due to the need to transmit data to a central server for processing.

Data privacy: In some industrial settings, the data generated by sensors may be sensitive and proprietary, and transmitting this data to a centralized server may raise data privacy concerns. Centralized machine learning approaches require that all data be sent to a central server for processing, which may not be acceptable in some environments.

Fault tolerance: Equipment failure or network disruptions can occur in decentralized settings, leading to data loss or service disruption. Traditional centralized machine-learning approaches may not be robust to these failures, leading to decreased system reliability and availability.

Why should we use decentralized Federated Learning?

Decentralized setting: In a decentralized setting, the data is distributed across multiple devices at the edge of the network, which can help to reduce the amount of data that needs to be transmitted to a central server. Decentralized settings can be particularly well-suited for on-device edge industrial networks, where data is generated locally on individual sensors or devices, and the communication resources are limited. By using federated learning in a decentralized setting, it is possible to train a model using data from multiple devices without transmitting it to a central server. This can help reduce communication costs and improve scalability. Additionally, in a decentralized setting, there is no single point of failure or security risk, as the data is distributed across multiple devices.

A decentralized setting best suits federated optimization in on-device edge industrial networks. It enables efficient and scalable learning using data from distributed edge devices while minimizing communication costs and reducing the risk of central server failure or security breaches. By using federated learning in a decentralized setting, it is possible to train models that perform well across multiple edge devices while still maintaining the privacy and security of the data generated on those devices.

On-device edge industrial networks: How are On-device edge industrial networks being used to improve the efficiency and reliability of industrial equipment. ?

On-device edge industrial networks are a type of network architecture in which data is processed and analyzed on the edge of the network, close to where it is generated, rather than being transmitted to a centralized server for processing.  This approach can help improve industrial equipment's efficiency and reliability by enabling real-time processing of sensor data, reducing communication bandwidth requirements, improving data privacy, and increasing fault tolerance.

Some examples of how on-device edge networks have been implemented in various industries:

Manufacturing: In manufacturing, on-device edge networks are used to monitor and optimize production processes in real-time. By processing sensor data locally and providing real-time feedback to operators and control systems, it is possible to improve the efficiency and quality of the production process.

Energy: In the energy industry, on-device edge networks are being used to monitor and optimize the performance of power plants and renewable energy systems. By processing sensor data locally and providing real-time feedback to operators, it is possible to reduce downtime, increase energy efficiency, and improve system reliability.

Transportation: In the transportation industry, on-device edge networks are being used to monitor and optimize the performance of vehicles and transportation infrastructure. By processing sensor data locally and providing real-time feedback to operators, it is possible to improve safety, reduce downtime, and optimize routes and schedules.

Research Objectives

My research aims to develop an on-device edge industrial network architecture, collect and preprocess sensor data/dataset, train and evaluate predictive maintenance models, compare the results to existing approaches, and identify opportunities for further research.

Develop an On-device edge industrial network architecture: The first objective of my research might be to design and develop an on-device edge industrial network architecture that is well-suited to the challenges of predictive maintenance in industrial settings. This architecture should consider the limitations of communication bandwidth and data privacy and provide a framework for implementing some federated Learning Algorithms.

Collect and preprocess sensor data/dataset: The second objective of my research might be to collect and preprocess sensor data from industrial equipment to train predictive maintenance models using federated optimization. This might involve developing tools and techniques for collecting data from a variety of different sources and processing the data to ensure that it is accurate and reliable.

Train and evaluate predictive maintenance models: The third objective of my research might be to train and evaluate predictive maintenance models using federated optimization and FL algorithm. This might involve developing custom machine learning models well-suited to the specific needs of industrial equipment maintenance and evaluating the performance of these models using various metrics.

Compare results to existing approaches: The fourth objective of my research might be to compare the results of my federated optimization approach to existing centralized machine learning approaches for predictive maintenance in industrial settings. This might involve benchmarking my approach against other approaches in the literature and identifying areas where my approach outperforms existing methods.

Identify opportunities for further research: The final objective of my research might be to identify opportunities for further research in federated optimization for on-device edge industrial networks. This might involve exploring new techniques and algorithms for federated optimization or developing new tools and technologies for collecting and processing sensor data in decentralized environments.

Significance of this research

The significance of my research lies in addressing the importance of predictive maintenance, the challenges associated with traditional approaches, the potential of federated optimization, the impact on on-device edge industrial networks, and the identification of future research opportunities. Specifically, this research aims to improve the reliability and efficiency of industrial equipment by utilizing the potential of federated optimization and the advantages of on-device edge industrial networks for predictive maintenance. Doing so has the potential to significantly reduce maintenance costs and improve operational efficiency, ultimately leading to cost savings and increased profits for industrial companies.

Reference:

1. Deng, M., Li, K., & Du, X. (2019). A predictive maintenance strategy based on machine learning for rotating machinery. Measurement, 131, 524-534. doi: 10.1016/j.measurement.2018.08.065
2. Mittal, A., Mahajan, P., & Kumar, R. (2016). A review on predictive maintenance and its impact on equipment downtime. International Journal of Engineering Research and General Science, 4(5), 118-125.
3. Park, J., Kim, S., Kim, H., & Kim, D. (2018). Predictive maintenance for semiconductor manufacturing equipment using machine learning. Journal of Intelligent Manufacturing, 29(4), 735-745. doi: 10.1007/s10845-016-1241-y
4. Prakash, R. K., & Rajamani, V. (2020). Review on predictive maintenance and reliability centered maintenance. Journal of Quality in Maintenance Engineering, 26(2), 185-200. doi: 10.1108/JQME-04-2019-0035
5. Parida, A., Kumar, U., Ribeiro, L. A. D., & Mukherjee, A. (2019). Predictive maintenance in industry 4.0: A review. Journal of Quality in Maintenance Engineering, 25(1), 5-42. doi.org/10.1108/JQME-05-2018-0042
6. Sankar, S., & Srinivasan, R. (2020). Predictive maintenance: A review. Journal of Quality in Maintenance Engineering, 26(1), 62-87. doi: 10.1108/JQME-06-2019-0056
7. Tao, F., Zuo, Y., Da Xu, L., & Zhang, L. (2014). IoT-based intelligent perception and access of manufacturing resource toward cloud manufacturing. IEEE Transactions on Industrial Informatics, 10(2), 1547-1557. doi: 10.1109/TII.2014.2300753
8. Wang, W., Hoang, D. T., Gong, M., & Niyato, D. (2020). Industrial big data analytics for predictive maintenance: A review. IEEE Transactions on Industrial Informatics, 16(9), 5714-5724. doi: 10.1109/TII.2019.2966803
9. Xue, X., Ma, Z., Wang, J., Liu, J., & Zhang, D. (2019). A predictive maintenance method based on wireless sensor networks and machine learning. International Journal of Distributed Sensor Networks, 15(8), 1550147719864700. doi: 10.1177/1550147719864700
10. Yang, S., Zhang, X., Zhang, Y., Wang, L., & Zhou, Z. (2020). A federated machine learning approach for predictive maintenance. Neurocomputing, 381, 232-244. doi: 10.1016/j.neucom.2019.11.045
11. Zhang, Z., Zhu, Y., Xing, L., Wang, F., Xie, L., & Wang, X. (2019). Industrial Internet of Things-based predictive maintenance system for machine tools. Journal of Manufacturing Systems, 51, 99-106. doi: 10.1016/j.jmsy.2019.02.002
12. Zhan, Q., Zhang, X., Chen, H., Liu, Y., & Chai, T. (2019). Predictive maintenance for a rolling mill using artificial intelligence techniques. Applied Sciences, 9(23), 5058. doi: 10.3390/app9235058
13. Source: Google Research Blog -https://ai.googleblog.com/2017/04/federated-learning-collaborative.html

Author: Romeo L Quoi Jr | Ph.D. Candidate | Harbin Institute of Technology