Share:


Computer vision based early fire-detection and firefighting mobile robots oriented for onsite construction

    Liulin Kong Affiliation
    ; Jichao Li Affiliation
    ; Shengyu Guo Affiliation
    ; Xiaojie Zhou Affiliation
    ; Di Wu Affiliation

Abstract

Fires are one of the most dangerous hazards and the leading cause of death in construction sites. This paper proposes a video-based firefighting mobile robot (FFMR), which is designed to patrol the desired territory and will constantly observe for fire-related events to make sure the camera without any occlusions. Once a fire is detected, the early warning system will send sound and light signals instantly and the FFMR moves to the right place to fight the fire source using the extinguisher. To improve the accuracy and speed of fire detection, an improved YOLOv3-Tiny (namely as YOLOv3-Tiny-S) model is proposed by optimizing its network structure, introducing a Spatial Pyramid Pooling (SPP) module, and refining the multi-scale anchor mechanism. The experiments show the proposed YOLOv3-Tiny-S model based FFMR can detect a small fire target with relatively higher accuracy and faster speed under the occlusions by outdoor environment. The proposed FFMR can be helpful to disaster management systems, avoiding huge ecological and economic losses, as well as saving a lot of human lives.

Keyword : convolutional neural network, firefighting, fire accidents prevention, mobile robot, improved YOLOv3-Tiny model, construction sites

How to Cite
Kong, L., Li, J., Guo, S., Zhou, X., & Wu, D. (2024). Computer vision based early fire-detection and firefighting mobile robots oriented for onsite construction. Journal of Civil Engineering and Management, 30(8), 720–737. https://doi.org/10.3846/jcem.2024.21360
Published in Issue
Oct 11, 2024
Abstract Views
628
PDF Downloads
284
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

Ahn, Y., Choi, H., & Kim, B. S. (2023). Development of early fire detection model for buildings using computer vision-based CCTV. Journal of Building Engineering, 65, Article 105647. https://doi.org/10.1016/j.jobe.2022.105647

Ando, H., Ambe, Y., Ishii, A., Konyo, M., Tadakuma, K., Maruyama, S., & Tadokoro, S. (2018). Aerial hose type robot by water jet for fire fighting. IEEE Robotics and Automation Letters, 3(2), 1128–1135. https://doi.org/10.1109/LRA.2018.2792701

Bosheng, S. (2022, November 25). Heavy construction fire accidents reported in 2020 (in Chinese). https://www.163.com/dy/article/GU0RDHF00552HS5R.html

Campbell, R. (2020). Fires in structures under construction or renovation. National Fire Protection Association.

CE Safety. (2022). Fires in buildings under construction caused by hot work. https://cesafety.co.uk/news/how-to-reduce-the-risk-of-fires-caused-by-hot-works/

Chang, C. K., Siagian, C., & Itti, L. (2010). Mobile robot vision navigation & localization using gist and saliency. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 4147–4154). IEEE.

Chaoxia, C., Shang, W., & Zhang, F. (2020). Information-guided flame detection based on faster r-cnn. IEEE Access, 8, 58923–58932. https://doi.org/10.1109/ACCESS.2020.2982994

Chen, C., Guorun, Y., Chenyu, W., Sotirios, G., & Shaohua, W. (2022). Enhancing the robustness of object detection via 6G vehicular edge computing. Digital Communications and Networks, 8(6), 923–931. https://doi.org/10.1016/j.dcan.2022.10.013

Chen, X., Xue, Y. P., Hou, Q. S., Fu, Y. & Zhu, Y. L. (2023). RepVGG-YOLOv7: A Modified YOLOv7 for fire smoke detection. Fire, 6(10), Article 383. https://doi.org/10.3390/fire6100383

Di Paola, D., Milella, A., Cicirelli, G., & Distante, A. (2010). An autonomous mobile robotic system for surveillance of indoor environments. International Journal of Advanced Robotic Systems, 7(1), 19–26. https://doi.org/10.5772/7254

Edirisinghe, R. (2019). Digital skin of the construction site: Smart sensor technologies towards the future smart construction site. Engineering, Construction and Architectural Management, 26(2), 184–223. https://doi.org/10.1108/ECAM-04-2017-0066

Fire Safety Matters. (2020). One-fifth of construction industry-related fires in England caused by hot work. https://www.fsmatters.com/Causes-of-construction-sector-fires-revealed

Geng, Y., Lai, M., Tian, X. C., Xu, X. L., Jiang, Y., & Zhang, Y. K. (2023). A novel seam extraction and path planning method for robotic welding of medium-thickness plate structural parts based on 3D vision. Robotics and Computer-Integrated Manufacturing, 79, Article 102433. https://doi.org/10.1016/j.rcim.2022.102433

Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1440–1448). IEEE. https://doi.org/10.1109/ICCV.2015.169

Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 580–587). IEEE. https://doi.org/10.1109/CVPR.2014.81

Han, X. F., Jin, J. S., Wang, M. J., Jiang, W., Gao, L., & Xiao, L. P. (2017). Video fire detection based on Gaussian Mixture Model and multi-color features. Signal, Image and Video Processing, 11(8), 1419–1425. https://doi.org/10.1007/s11760-017-1102-y

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1904–1916. https://doi.org/10.1109/TPAMI.2015.2389824

He, W., Huang, Z., Wei, Z., Li, C., & Guo, B. (2019). TF-YOLO: An improved incremental network for real-time object detection. Applied Sciences, 9(16), Article 3225. https://doi.org/10.3390/app9163225

Hefeeda, M., & Bagheri, M. (2007, October). Wireless sensor networks for early detection of forest fires. In 2007 IEEE International Conference on Mobile Adhoc and Sensor Systems. IEEE. https://doi.org/10.1109/MOBHOC.2007.4428702

Hong, X., Wang, W., & Liu, Q. (2019, June). Design and realization of fire detection using computer vision technology. In 2019 Chinese Control and Decision Conference (CCDC) (pp. 5645–5649). IEEE. https://doi.org/10.1109/CCDC.2019.8832897

Innocente, M. S., & Grasso, P. (2019). Self-organising swarms of firefighting drones: Harnessing the power of collective intelligence in decentralised multi-robot systems. Journal of Computational Science, 34, 80–101. https://doi.org/10.1016/j.jocs.2019.04.009

Jiao, Z., Zhang, Y., Xin, J., Mu, L., Yi, Y., Liu, H., & Liu, D. (2019, July). A deep learning based forest fire detection approach using UAV and YOLOv3. In 2019 1st International Conference on Industrial Artificial Intelligence (IAI). IEEE. https://doi.org/10.1109/ICIAI.2019.8850815

Kim, J. H., Jo, S., & Lattimer, B. Y. (2016). Feature selection for intelligent firefighting robot classification of fire, smoke, and thermal reflections using thermal infrared images. Journal of Sensors, 2016, Article 8410731. https://doi.org/10.1155/2016/8410731

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386

Li, H., Lu, M., Hsu, S. C., Gray, M., & Huang, T. (2015). Proactive behavior-based safety management for construction safety improvement. Safety Science, 75, 107–117. https://doi.org/10.1016/j.ssci.2015.01.013

Li, S., Feng, C., Niu, Y., Shi, L., Wu, Z., & Song, H. (2019). A fire reconnaissance robot based on SLAM position, thermal imaging technologies, and AR display. Sensors, 19(22), Article 5036. https://doi.org/10.3390/s19225036

Li, Z., Mihaylova, L., & Yang, L. (2021). A deep learning framework for autonomous flame detection. Neurocomputing, 448, 205–216. https://doi.org/10.1016/j.neucom.2021.03.019

Li, S., Wang, Y. H., Feng, C. Y., Zhang, D., Li, H. Z., Huang, W. & Shi, L. (2022). A thermal imaging flame-detection model for firefighting robot based on YOLOv4-F model. Fire, 5(5), Article 172. https://doi.org/10.3390/fire5050172

Liu, S., Tu, D., & Zhang, Y. (2009, November). Multiparameter fire detection based on wireless sensor network. In 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems (Vol. 3, pp. 203–206). IEEE. https://doi.org/10.1109/ICICISYS.2009.5358197

Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. In B. Leibe, J. Matas, N. Sebe, & M. Welling (Eds.), Lecture notes in computer science: Vol. 9905. Computer vision – ECCV 2016 (pp. 21–37). Springer, Cham. https://doi.org/10.1007/978-3-319-46448-0_2

Madhevan, B., Sakkaravarthi, R., Singh, G. M., Diya, R., & Jha, D. K. (2017). Modelling, simulation and mechatronics design of a wireless automatic fire fighting surveillance robot. Defence Science Journal, 67(5), Article 572. https://doi.org/10.14429/dsj.67.10237

Management & Emergency. (2020). Nationalwide fire and fire rescue reported in 2020 (in Chinese). https://www.119.gov.cn/article/3xBeEJjR54K

McNeil, J. G., & Lattimer, B. Y. (2017). Robotic fire suppression through autonomous feedback control. Fire Technology, 53(3), 1171–1199. https://doi.org/10.1007/s10694-016-0623-1

Mir-Nasiri, N., Siswoyo J. H., & Ali, M. H. (2018). Portable autonomous window cleaning robot. Procedia Computer Science, 133, 197–204. https://doi.org/10.1016/j.procs.2018.07.024

Muhammad, K., Ahmad, J., & Baik, S. W. (2018). Early fire detection using convolutional neural networks during surveillance for effective disaster management. Neurocomputing, 288, 30–42. https://doi.org/10.1016/j.neucom.2017.04.083

Nguyen, A. Q., Nguyen, H. T., Tran, V. C., Pham, H. X. & Pestana, J. (2021). A visual real-time fire detection using single shot MultiBox detector for UAV-based fire surveillance. In 8th IEEE International Conference on Communications and Electronics (IEEE ICCE), Vietnam. https://doi.org/10.1109/ICCE48956.2021.9352080

Palmer, A. (2012). Hotel construction site catches fire. Top Stories, Midland Reporter-Telegram. Texas.

Park, J., Cho, Y. K., & Martinez, D. (2016). A BIM and UWB integrated mobile robot navigation system for indoor position tracking applications. Journal of Construction Engineering and Project Management, 6(2), 30–39. https://doi.org/10.6106/JCEPM.2016.6.2.030

Pincott, J., Tien, P. W., Wei, S., & Calautit, J. K. (2022). Development and evaluation of a vision-based transfer learning approach for indoor fire and smoke detection. Building Services Engineering Research and Technology, 43, 319–332. https://doi.org/10.1177/01436244221089445

Prema, C. E., Vinsley, S., & Suresh, S. (2018). Efficient flame detection based on static and dynamic texture analysis in forest fire detection. Fire Technology, 54(1), 255–288. https://doi.org/10.1007/s10694-017-0683-x

Qiu, X., Xi, T., Sun, D., Zhang, E., Li, C., Peng, Y., Wei, J., & Wang, G. (2018). Fire detection algorithm combined with image processing and flame emission spectroscopy. Fire Technology, 54(5), 1249–1263. https://doi.org/10.1007/s10694-018-0727-x

Redmon, J., & Farhadi, A. (2017). YOLO9000: better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7263–7271). IEEE. https://doi.org/10.1109/CVPR.2017.690

Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv. https://doi.org/10.48550/arXiv.1804.02767

Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 779–788). IEEE. https://doi.org/10.1109/CVPR.2016.91

Ren, S., He, K., Girshick, R., & Sun, J. (2016). Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

Roberto, G. F., Branco, K. C., Machado, J. M., & Pinto, A. R. (2013, April). Local data fusion algorithm for fire detection through mobile robot. In 2013 14th Latin American Test Workshop-LATW. IEEE. https://doi.org/10.1109/LATW.2013.6562667

Shen, D., Chen, X., Nguyen, M., & Yan, W. Q. (2018, April). Flame detection using deep learning. In 2018 4th International conference on control, automation and robotics (ICCAR) (pp. 416–420). IEEE. https://doi.org/10.1109/ICCAR.2018.8384711

So, A. T., Lo, T. Y., & Chan, W. L. (1996). An autonomous robotic cladding inspector for high-rise buildings in Hong Kong. HKIE Transactions, 3(2), 37–45. https://doi.org/10.1080/1023697X.1996.10667701

Sridhar, P., & Sathiya, R. (2021). Computer vision based early electrical fire-detection in video surveillance oriented for building environment. Journal of Physics: Conference Series, 1916, Article 012024. https://doi.org/10.1088/1742-6596/1916/1/012024

Su, Y., Mao, C., Jiang, R., Liu, G., & Wang, J. (2021). Data-driven fire safety management at building construction sites: Leveraging CNN. Journal of Management in Engineering, 37(2), Article 04020108. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000877

Tesema, F. B., Lin, J., Ou, J., Wu, H., & Zhu, W. (2018). Feature fusing of feature pyramid network for multi-scale pedestrian detection. In 2018 15th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) (pp. 10–13). IEEE. https://doi.org/10.1109/ICCWAMTIP.2018.8632614

Tso, S. K., & Feng, T. O. N. G. (2003, September). Robot assisted wall inspection for improved maintenance of high-rise buildings. In 20th International Symposium on Automation and Robotics in Construction (pp. 449–455). https://doi.org/10.22260/ISARC2003/0071

Victores, J. G., Martínez, S., Jardón, A., & Balaguer, C. (2011). Robot-aided tunnel inspection and maintenance system by vision and proximity sensor integration. Automation in Construction, 20(5), 629–636. https://doi.org/10.1016/j.autcon.2010.12.005

Vishaal, R., Raghavan, P., Rajesh, R., Michael, S., & Elara, M. R. (2018). Design of dual purpose cleaning robot. Procedia Computer Science, 133, 518–525. https://doi.org/10.1016/j.procs.2018.07.065

Wang, Y., Xing, J. P., Guo, H., & Wang, L. J. (2017). Key technologies of tunnel firefighting robots. IETE Technical Review, 34(1), 3–10. https://doi.org/10.1080/02564602.2016.1139475

Wang, Z., Li, H., & Zhang, X. (2019). Construction waste recycling robot for nails and screws: Computer vision technology and neural network approach. Automation in Construction, 97, 220–228. https://doi.org/10.1016/j.autcon.2018.11.009

Wang, Z., Li, H., & Yang, X. (2020). Vision-based robotic system for on-site construction and demolition waste sorting and recycling. Journal of Building Engineering, 32, Article 101769. https://doi.org/10.1016/j.jobe.2020.101769

Wu, H., Wu, D., & Zhao, J. (2019). An intelligent fire detection approach through cameras based on computer vision methods. Process Safety and Environmental Protection, 127, 245–256. https://doi.org/10.1016/j.psep.2019.05.016

Wu, Z., Xue, R., & Li, H. (2022). Real-time video fire detection via modified YOLOv5 network mode. Fire Technology, 58(4), 2377–2403. https://doi.org/10.1007/s10694-022-01260-z

Xiao, D., Shan, F., Li, Z., Le, B.T., Liu, X., & Li, X. (2019). A target detection model based on improved Tiny-Yolov3 under the environment of mining truck. IEEE Access, 7, 123757–123764. https://doi.org/10.1109/ACCESS.2019.2928603

Xie, Y., Zhu, J., Cao, Y., Zhang, Y., Feng, D., Zhang, Y., & Chen, M. (2020). Efficient video fire detection exploiting motion-flicker-based dynamic features and deep static features. IEEE Access, 8, 81904–81917. https://doi.org/10.1109/ACCESS.2020.2991338

Xu, Z., Guo, Y., & Saleh, J. H. (2020). Tackling small data challenges in visual fire detection: A deep convolutional generative adversarial network approach. IEEE Access, 9, 3936–3946. https://doi.org/10.1109/ACCESS.2020.3047764

Yang, H., Jang, H., Kim, T., & Lee, B. (2019). Non-temporal lightweight fire detection network for intelligent surveillance systems. IEEE Access, 7, 169257–169266. https://doi.org/10.1109/ACCESS.2019.2953558

Yar, H., Khan, Z. A., Ullah, F. U. M., Ullah, W. & Baik, S. W. (2023). A modified YOLOv5 architecture for efficient fire detection in smart cities. Expert Systems with Applications, 231, Article 120465. https://doi.org/10.1016/j.eswa.2023.120465

Yi, Z., Yongliang, S., & Jun, Z. (2019). An improved TINY-YOLOV3 pedestrian detection algorithm. Optik, 183, 17–23. https://doi.org/10.1016/j.ijleo.2019.02.038

Zhan, H. W., Pei, X. Y., Zhang, T. H. & Zhang, L. Q. (2023). Research on flame detection method based on improved SSD algorithm. Journal of Intelligent & Fuzzy Systems, 45, 6501–6512. https://doi.org/10.3233/JIFS-232645

Zhang, J., Jin, Z., & Feng, H. (2018). Type synthesis of a 3-mixed-DOF protectable leg mechanism of a firefighting multi-legged robot based on GF set theory. Mechanism and Machine Theory, 130, 567–584. https://doi.org/10.1016/j.mechmachtheory.2018.08.026

Zhang, H., Qin, L., Li, J., Guo, Y., Zhou, Y., Zhang, J., & Xu, Z. (2020). Real-time detection method for small traffic signs based on Yolov3. IEEE Access, 8, 64145–64156. https://doi.org/10.1109/ACCESS.2020.2984554

Zhang, H., Wang, Z., Chen, M., Peng, Y., Gao, Y., & Zhou, J. (2021). An improved YOLOv3 algorithm combined with attention mechanism for flame and smoke detection. In X. Sun, X. Zhang, Z. Xia, & E. Bertino (Eds.), Lecture notes in computer science: Vol. 12736. Artificial intelligence and security. ICAIS 2021 (pp. 226–238). Springer, Cham. https://doi.org/10.1007/978-3-030-78609-0_20

Zhang, T., Wang, Z., Zeng, Y., Wu, X., Huang, X., & Xiao, F. (2022). Building artificial-intelligence digital fire (AID-Fire) system: A real-scale demonstration. Journal of Building Engineering, 62, Article 105363. https://doi.org/10.1016/j.jobe.2022.105363

Zhu, J., Li, W., Lin, D., Cheng, H., & Zhao, G. (2020a). Intelligent fire monitor for fire robot based on infrared image feedback control. Fire Technology, 56(5), 2089–2109. https://doi.org/10.1007/s10694-020-00964-4

Zhu, J., Pan, L., & Zhao, G. (2020b). An improved near-field computer vision for jet trajectory falling position prediction of intelligent fire robot. Sensors, 20(24), Article 7029. https://doi.org/10.3390/s20247029