Probabilistic management of pavement defects with image processing techniques
Abstract
Pavement management has traditionally relied on human-based decisions. In many countries, however, the pavement stock has recently increased, while the number of management experts has declined, posing the challenge of how to efficiently manage the larger stock with fewer resources. Compared to efficient computer-based techniques, human-based methods are more prone to errors that compromise analysis and decisions. This research built a robust probabilistic pavement management model with a safety metric output using inputs from image processing tested against the judgment of experts. The developed model optimized road pavement safety. The study explored image processing techniques considering the trade-off between processing cost and output accuracy, with annotation precision and intersection over union (IoU) set objectively. The empirical applicability of the model is shown for selected roads in Japan.
Keyword : pavement management, image processing, objective annotation and IoU, expert validation
How to Cite
Obunguta, F., Matsushima, K., & Susaki, J. (2024). Probabilistic management of pavement defects with image processing techniques. Journal of Civil Engineering and Management, 30(2), 114–132. https://doi.org/10.3846/jcem.2024.20401
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Arya, D., Maeda, H., Ghosh, S. K., Toshniwal, D., Mraz, A., Kashiyama, T., & Sekimoto, Y. (2020a). Transfer learning-based road damage detection for multiple countries. arXiv:2008.13101. https://doi.org/10.48550/arXiv.2008.13101
Arya, D., Maeda, H., Ghosh, S. K., Toshniwal, D., Mraz, A., Kashiyama, T., & Sekimoto, Y. (2020b). Global road damage detection: state-of-the-art solutions. In IEEE International Conference on Big Data (Big Data) (pp. 5533–5539). https://doi.org/10.1109/BigData50022.2020.9377790
Arya, D., Maeda, H., Ghosh, S. K., Toshniwal, D., Mraz, A., Kashiyama, T., & Sekimoto, Y. (2021). Deep learning-based road damage detection and classification for multiple countries. Automation in Construction, 132, Article 103935. https://doi.org/10.1016/j.autcon.2021.103935
Barchard, K. A., & Pace, L. A. (2011). Preventing human error: The impact of data entry methods on data accuracy and statistical results. Computers in Human Behavior, 27(5), 1834–1839. https://doi.org/10.1016/j.chb.2011.04.004
Bosurgi, G., Modica, M., Pellegrino, O., & Sollazzo, G. (2022). An automatic pothole detection algorithm using pavement 3D data. International Journal of Pavement Engineering. https://doi.org/10.1080/10298436.2022.2057978
Cambridge University. (2021). CamVid dataset. http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/
Deng, J., Dong, W., Socher, R., Li, L. -J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 248–255), Florida, USA. https://doi.org/10.1109/CVPR.2009.5206848
Everingham, M., Eslami, S. A., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2015). The Pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1), 98–136. https://doi.org/10.1007/s11263-014-0733-5
Georgopoulos, A., Loizos, A., & Flouda, A. (1995). Digital image processing as a tool for pavement distress evaluation. ISPRS Journal of Photogrammetry and Remote sensing, 50(1), 23–33. https://doi.org/10.1016/0924-2716(95)91844-A
Girshick, R. (2015). Fast R-CNN. In IEEE International Conference on Computer Vision (pp. 1440–1448), Boston, MA, USA. https://doi.org/10.1109/ICCV.2015.169
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 580–587), Ohio, USA. https://doi.org/10.1109/CVPR.2014.81
Goncalves, L. R., & Givigi, S. N. (2016). Automatic crack detection and measurement based on image analysis. IEEE Transactions on Instrumentation and Measurement, 65(3), 583–590. https://doi.org/10.1109/TIM.2015.2509278
Greenwald, N. F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Fullaway, C. C., McIntosh, B. J., Leow, K. X., Schwartz, M. S., Pavelchek, C., Cui, S., Camplisson, I., Bar-Tal, O., Singh, J., Fong, M., Chaudhry, G., Abraham, Z., Moseley, J., … Van Valen, D. (2022). Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature Biotechnology, 40, 555–565. https://doi.org/10.1038/s41587-021-01094-0
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2018). Mask R-CNN, Facebook AI Research (FAIR). arXiv:1703.06870. https://doi.org/10.48550/arXiv.1703.06870
Hong, J. W., Jin, S., & Lee, S. E. (2020). A vision-based approach for autonomous crack width measurement with flexible kernel. Automation in Construction, 110, Article 103019. https://doi.org/10.1016/j.autcon.2019.103019
JARA. (2013). Maintenance and repair guide book of the pavement 2013 (1st ed.). Japan Road Association, Tokyo.
Kerali, H. G. (2000). HDM-4: Highway development and management. Volume one: overview of HDM-4.
Kinyanjui, N. M., Odonga, T., Cintas, C., Codella, N. C. F., Panda, R., Sattigeri, P., & Varshney, K. R. (2019). Estimating skin tone and effects on classification performance in dermatology datasets. In NeurIPS Workshop on Fair ML for Health, Vancouver, Canada. https://doi.org/10.48550/arXiv.1910.13268
Kobayashi, K., Do, M., & Han, D. (2010). Estimation of Markovian transition probabilities for pavement deterioration forecasting. KSCE Journal of Civil Engineering, 14(3), 343–351. https://doi.org/10.1007/s12205-010-0343-x
Kobayashi, K., Kaito, K., & Lethanh, N. (2012). A Bayesian estimation method to improve deterioration prediction for infrastructure system with Markov chain model. International Journal of Architecture, Engineering and Construction, 1(1), 1–13. https://doi.org/10.7492/IJAEC.2012.001
Kobayashi, K., Eguchi, M., Oi, A., Aoki, K., & Kaito, K. (2013). The optimal implementation policy for inspecting pavement with deterioration uncertainty. Journal of Japan Society of Civil Engineers, 1(1), 551–568. https://doi.org/10.2208/journalofjsce.1.1_551
Kroon, D. (2021). Region growing. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/19084-region-growing
Kubo, K. (2017). Pavement maintenance in Japan. In Road Conference International Symposium. https://www.road.or.jp/international/pdf/32_AM6.pdf
Lethanh, N., & Adey, B. T. (2012). A hidden Markov model for modeling pavement deterioration under incomplete monitoring data. International Journal of Civil and Environmental Engineering, 6(1). https://doi.org/10.5281/zenodo.1082516
Lin, K., & Lin, C. (2011). Applying utility theory to cost allocation of pavement maintenance and repair. International Journal of Pavement Research and Technology, 4(4), 212–221.
Liu, H., & Wang, D. Z. W. (2016). Modeling and solving discrete network design problem with stochastic user equilibrium. Journal of Advanced Transportation, 50(7), 1295–1313. https://doi.org/10.1002/atr.1402
Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., & Omata, H. (2018). Road damage detection and classification using deep neural networks with smartphone images. Journal of Computer-Aided Civil and Infrastructure Engineering, 33, 1127–1141. https://doi.org/10.1111/mice.12387
Minami, M., & Suzuki, T. (2008). Pavement maintenance level and annual budget over the regional road network. Journal of Construction Management, 15, 71–79. https://doi.org/10.2208/procm.15.71
Mirikharaji, Z., Abhishek, K., Izadi, S., & Hamarneh, G. (2021). D-LEMA: Deep learning ensembles from multiple annotations application to skin lesion segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 1837–1846). https://doi.org/10.1109/CVPRW53098.2021.00203
Miyamoto, A., & Yoshitake, T. (2009). Development of a remote condition assessment system for road infrastructures. In International ECCE Conference, EUROINFRA 2009, Current State and Challenges for Sustainable Development of Infrastructure. https://www.irbnet.de/daten/iconda/CIB17687.pdf
Mizutani, D., Nakazato, Y., & Lee, J. (2020). Network-level synchronized pavement repair and work zone policies: Optimal solution and rule-based approximation. Transportation Research Part C: Emerging Technologies, 120, Article 102797. https://doi.org/10.1016/j.trc.2020.102797
Obunguta, F., & Matsushima, K. (2020). Optimal pavement management strategy development with a stochastic model and its practical application to Ugandan national roads. International Journal of Pavement Engineering, 23(7), 2405–2419. https://doi.org/10.1080/10298436.2020.1857759
Obunguta, F., Matsushima, K., & Bakamwesiga, H. (2022). Social cost optimization model and empirical evaluation of intervention effects on Ugandan road pavements. Journal of Infrastructure Systems, 28(4), Article 05022005. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000707
Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62–66. https://doi.org/10.1109/TSMC.1979.4310076
Padilla, R., Netto, S. L., & da Silva, E. A. B. (2020). A survey on performance metrics for object-detection algorithms. In Proceedings of the International Conference on Systems, Signals and Image Processing (IWSSIP) (pp. 237–242). https://doi.org/10.1109/IWSSIP48289.2020.9145130
Pérez-Acebo, H., Mindrab, N., Railean, A., & Rojí, E. (2019). Rigid pavement performance models by means of Markov Chains with half-year step time. International Journal of Pavement Engineering, 20, 830–843. https://doi.org/10.1080/10298436.2017.1353390
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. arXiv:1506.01497. https://doi.org/10.48550/arXiv.1506.01497
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592. https://doi.org/10.1093/biomet/63.3.581
Rubin, D. B. (1987). Multiple imputation for non-response in surveys. John Wiley. https://doi.org/10.1002/9780470316696
Tabatabaee, N., & Ziyadi, M. (2013). Bayesian approach to updating Markov-based models for predicting pavement performance. Transportation Research Record: Journal of the Transportation Research Board, 2366(1), 34–42. https://doi.org/10.3141/2366-04
Thao, N. D., Aoki, K., Kato, T., Toan, T. N., Kobayashi, K., & Kaito, K. (2015). A practical process to introduce a customized pavement management system in Vietnam. Journal of Japan Society of Civil Engineers, 3(1), 246–258. https://doi.org/10.2208/journalofjsce.3.1_246
The MathWorks Inc. (2021). https://uk.mathworks.com/help/images/ref/lazysnapping.html
Thuyet, D. Q., Jomoto, M., Hirakawa, K., & Lei Swe, Y. L. (2022). Development of an autonomous road surface damage inspection program using deep convolutional neural network. Journal of Japan Society of Civil Engineers, 10(1), 235–246. https://doi.org/10.2208/journalofjsce.10.1_235
Tsuda, T., Kaito, K., Aoki, K., & Kobayashi, K. (2006). Estimating Markovian transition probabilities for bridge deterioration forecasting. Structural Engineering/Earthquake Engineering, 23(2), 241s–256s. https://doi.org/10.2208/jsceseee.23.241s
Xu, Y., Zhong, Z., Lian, D., Li, J., Li, Z., Xu, X., & Gao, S. (2021). Crowd counting with partial annotations in an image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 15570–15579). https://doi.org/10.1109/ICCV48922.2021.01528
Yoshida, T. (2016). Composite indicators for assessing maintenance needs for road pavements from the view point of road functions. Journal of Japan Society of Civil Engineers, Ser. E1 (Pavement Engineering), 72(1), 12–20. https://doi.org/10.2208/jscejpe.72.12
Zou, D., Zhang, M., Bai, Z., Liu, T., Zhou, A., Wang, X., Cui, W., & Zhang, S. (2022). Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Journal of Computer-Aided Civil and Infrastructure Engineering, 37(9), 1188–1204. https://doi.org/10.1111/mice.12815
Arya, D., Maeda, H., Ghosh, S. K., Toshniwal, D., Mraz, A., Kashiyama, T., & Sekimoto, Y. (2020b). Global road damage detection: state-of-the-art solutions. In IEEE International Conference on Big Data (Big Data) (pp. 5533–5539). https://doi.org/10.1109/BigData50022.2020.9377790
Arya, D., Maeda, H., Ghosh, S. K., Toshniwal, D., Mraz, A., Kashiyama, T., & Sekimoto, Y. (2021). Deep learning-based road damage detection and classification for multiple countries. Automation in Construction, 132, Article 103935. https://doi.org/10.1016/j.autcon.2021.103935
Barchard, K. A., & Pace, L. A. (2011). Preventing human error: The impact of data entry methods on data accuracy and statistical results. Computers in Human Behavior, 27(5), 1834–1839. https://doi.org/10.1016/j.chb.2011.04.004
Bosurgi, G., Modica, M., Pellegrino, O., & Sollazzo, G. (2022). An automatic pothole detection algorithm using pavement 3D data. International Journal of Pavement Engineering. https://doi.org/10.1080/10298436.2022.2057978
Cambridge University. (2021). CamVid dataset. http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/
Deng, J., Dong, W., Socher, R., Li, L. -J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 248–255), Florida, USA. https://doi.org/10.1109/CVPR.2009.5206848
Everingham, M., Eslami, S. A., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2015). The Pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1), 98–136. https://doi.org/10.1007/s11263-014-0733-5
Georgopoulos, A., Loizos, A., & Flouda, A. (1995). Digital image processing as a tool for pavement distress evaluation. ISPRS Journal of Photogrammetry and Remote sensing, 50(1), 23–33. https://doi.org/10.1016/0924-2716(95)91844-A
Girshick, R. (2015). Fast R-CNN. In IEEE International Conference on Computer Vision (pp. 1440–1448), Boston, MA, USA. https://doi.org/10.1109/ICCV.2015.169
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 580–587), Ohio, USA. https://doi.org/10.1109/CVPR.2014.81
Goncalves, L. R., & Givigi, S. N. (2016). Automatic crack detection and measurement based on image analysis. IEEE Transactions on Instrumentation and Measurement, 65(3), 583–590. https://doi.org/10.1109/TIM.2015.2509278
Greenwald, N. F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Fullaway, C. C., McIntosh, B. J., Leow, K. X., Schwartz, M. S., Pavelchek, C., Cui, S., Camplisson, I., Bar-Tal, O., Singh, J., Fong, M., Chaudhry, G., Abraham, Z., Moseley, J., … Van Valen, D. (2022). Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature Biotechnology, 40, 555–565. https://doi.org/10.1038/s41587-021-01094-0
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2018). Mask R-CNN, Facebook AI Research (FAIR). arXiv:1703.06870. https://doi.org/10.48550/arXiv.1703.06870
Hong, J. W., Jin, S., & Lee, S. E. (2020). A vision-based approach for autonomous crack width measurement with flexible kernel. Automation in Construction, 110, Article 103019. https://doi.org/10.1016/j.autcon.2019.103019
JARA. (2013). Maintenance and repair guide book of the pavement 2013 (1st ed.). Japan Road Association, Tokyo.
Kerali, H. G. (2000). HDM-4: Highway development and management. Volume one: overview of HDM-4.
Kinyanjui, N. M., Odonga, T., Cintas, C., Codella, N. C. F., Panda, R., Sattigeri, P., & Varshney, K. R. (2019). Estimating skin tone and effects on classification performance in dermatology datasets. In NeurIPS Workshop on Fair ML for Health, Vancouver, Canada. https://doi.org/10.48550/arXiv.1910.13268
Kobayashi, K., Do, M., & Han, D. (2010). Estimation of Markovian transition probabilities for pavement deterioration forecasting. KSCE Journal of Civil Engineering, 14(3), 343–351. https://doi.org/10.1007/s12205-010-0343-x
Kobayashi, K., Kaito, K., & Lethanh, N. (2012). A Bayesian estimation method to improve deterioration prediction for infrastructure system with Markov chain model. International Journal of Architecture, Engineering and Construction, 1(1), 1–13. https://doi.org/10.7492/IJAEC.2012.001
Kobayashi, K., Eguchi, M., Oi, A., Aoki, K., & Kaito, K. (2013). The optimal implementation policy for inspecting pavement with deterioration uncertainty. Journal of Japan Society of Civil Engineers, 1(1), 551–568. https://doi.org/10.2208/journalofjsce.1.1_551
Kroon, D. (2021). Region growing. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/19084-region-growing
Kubo, K. (2017). Pavement maintenance in Japan. In Road Conference International Symposium. https://www.road.or.jp/international/pdf/32_AM6.pdf
Lethanh, N., & Adey, B. T. (2012). A hidden Markov model for modeling pavement deterioration under incomplete monitoring data. International Journal of Civil and Environmental Engineering, 6(1). https://doi.org/10.5281/zenodo.1082516
Lin, K., & Lin, C. (2011). Applying utility theory to cost allocation of pavement maintenance and repair. International Journal of Pavement Research and Technology, 4(4), 212–221.
Liu, H., & Wang, D. Z. W. (2016). Modeling and solving discrete network design problem with stochastic user equilibrium. Journal of Advanced Transportation, 50(7), 1295–1313. https://doi.org/10.1002/atr.1402
Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., & Omata, H. (2018). Road damage detection and classification using deep neural networks with smartphone images. Journal of Computer-Aided Civil and Infrastructure Engineering, 33, 1127–1141. https://doi.org/10.1111/mice.12387
Minami, M., & Suzuki, T. (2008). Pavement maintenance level and annual budget over the regional road network. Journal of Construction Management, 15, 71–79. https://doi.org/10.2208/procm.15.71
Mirikharaji, Z., Abhishek, K., Izadi, S., & Hamarneh, G. (2021). D-LEMA: Deep learning ensembles from multiple annotations application to skin lesion segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 1837–1846). https://doi.org/10.1109/CVPRW53098.2021.00203
Miyamoto, A., & Yoshitake, T. (2009). Development of a remote condition assessment system for road infrastructures. In International ECCE Conference, EUROINFRA 2009, Current State and Challenges for Sustainable Development of Infrastructure. https://www.irbnet.de/daten/iconda/CIB17687.pdf
Mizutani, D., Nakazato, Y., & Lee, J. (2020). Network-level synchronized pavement repair and work zone policies: Optimal solution and rule-based approximation. Transportation Research Part C: Emerging Technologies, 120, Article 102797. https://doi.org/10.1016/j.trc.2020.102797
Obunguta, F., & Matsushima, K. (2020). Optimal pavement management strategy development with a stochastic model and its practical application to Ugandan national roads. International Journal of Pavement Engineering, 23(7), 2405–2419. https://doi.org/10.1080/10298436.2020.1857759
Obunguta, F., Matsushima, K., & Bakamwesiga, H. (2022). Social cost optimization model and empirical evaluation of intervention effects on Ugandan road pavements. Journal of Infrastructure Systems, 28(4), Article 05022005. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000707
Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62–66. https://doi.org/10.1109/TSMC.1979.4310076
Padilla, R., Netto, S. L., & da Silva, E. A. B. (2020). A survey on performance metrics for object-detection algorithms. In Proceedings of the International Conference on Systems, Signals and Image Processing (IWSSIP) (pp. 237–242). https://doi.org/10.1109/IWSSIP48289.2020.9145130
Pérez-Acebo, H., Mindrab, N., Railean, A., & Rojí, E. (2019). Rigid pavement performance models by means of Markov Chains with half-year step time. International Journal of Pavement Engineering, 20, 830–843. https://doi.org/10.1080/10298436.2017.1353390
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. arXiv:1506.01497. https://doi.org/10.48550/arXiv.1506.01497
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592. https://doi.org/10.1093/biomet/63.3.581
Rubin, D. B. (1987). Multiple imputation for non-response in surveys. John Wiley. https://doi.org/10.1002/9780470316696
Tabatabaee, N., & Ziyadi, M. (2013). Bayesian approach to updating Markov-based models for predicting pavement performance. Transportation Research Record: Journal of the Transportation Research Board, 2366(1), 34–42. https://doi.org/10.3141/2366-04
Thao, N. D., Aoki, K., Kato, T., Toan, T. N., Kobayashi, K., & Kaito, K. (2015). A practical process to introduce a customized pavement management system in Vietnam. Journal of Japan Society of Civil Engineers, 3(1), 246–258. https://doi.org/10.2208/journalofjsce.3.1_246
The MathWorks Inc. (2021). https://uk.mathworks.com/help/images/ref/lazysnapping.html
Thuyet, D. Q., Jomoto, M., Hirakawa, K., & Lei Swe, Y. L. (2022). Development of an autonomous road surface damage inspection program using deep convolutional neural network. Journal of Japan Society of Civil Engineers, 10(1), 235–246. https://doi.org/10.2208/journalofjsce.10.1_235
Tsuda, T., Kaito, K., Aoki, K., & Kobayashi, K. (2006). Estimating Markovian transition probabilities for bridge deterioration forecasting. Structural Engineering/Earthquake Engineering, 23(2), 241s–256s. https://doi.org/10.2208/jsceseee.23.241s
Xu, Y., Zhong, Z., Lian, D., Li, J., Li, Z., Xu, X., & Gao, S. (2021). Crowd counting with partial annotations in an image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 15570–15579). https://doi.org/10.1109/ICCV48922.2021.01528
Yoshida, T. (2016). Composite indicators for assessing maintenance needs for road pavements from the view point of road functions. Journal of Japan Society of Civil Engineers, Ser. E1 (Pavement Engineering), 72(1), 12–20. https://doi.org/10.2208/jscejpe.72.12
Zou, D., Zhang, M., Bai, Z., Liu, T., Zhou, A., Wang, X., Cui, W., & Zhang, S. (2022). Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Journal of Computer-Aided Civil and Infrastructure Engineering, 37(9), 1188–1204. https://doi.org/10.1111/mice.12815