Iranian Journal of Remote Sensing and GIS

Iranian Journal of Remote Sensing and GIS

Performance Analysis of Support Vector Machine, Random Forest, and Maximum Likelihood Algorithms in Land Use Classification of the Metropolitan Area of Mashhad

Document Type : Original Article

Authors
Sajedeh Baghban
Mohammad Rahim Rahnama
Mohammad Ajza Shokuhi
Hossein Vahidi
Dep of Geography, Faculty of Literature and Human Sciences, Ferdowsi University of Mashhad, Mashhad, Iran
10.48308/gisj.2024.236864.1230
Abstract
Introduction: Considering that the value and usability of any map produced from satellite images depend on its accuracy, evaluating the accuracy of satellite image classification methods is of great importance. Therefore, this research aims to analyse the performance of Support Vector Machine (SVM), Random Forest (RF), and Maximum Likelihood Classification (MLC) algorithms in identifying land use and land cover (LULC) in the metropolitan area of Mashhad. Numerous algorithms have been developed for satellite image classification to date, and their performance varies under different conditions. For this reason, this study first identifies the most commonly used algorithms through a review of previous research, and then, by assessing the characteristics of various classifiers, selects the three algorithms: Support Vector Machine, Random Forest, and Maximum Likelihood. There are various studies regarding the performance of different classification algorithms, each yielding different results. Given that multiple studies have shown that LULC mapping accuracy is related to time and location, and that each of these studies has emphasized the accuracy of different algorithms, their results cannot be generalized to the geographical conditions of Iran. On the other hand, there has not been sufficient research in the geomorphological conditions of Iran to assess the accuracy of classification algorithms, and most studies validating these algorithms have been conducted in case studies outside of Iran. Therefore, considering the differences in algorithm results under various conditions, examining the accuracy and performance of these algorithms focusing on the extensive and diverse metropolitan area of Mashhad may yield novel and noteworthy findings.
Materials and Methods: The present research is applied in terms of purpose and descriptive-analytical in terms of nature. Data collection in this study has been conducted through a documentary-library method. In this study, images from the OLI sensor on the Landsat 8 satellite were used. The classification of satellite images was performed in two stages: preprocessing and processing. After assessing the accuracy of the classification using the Kappa coefficient, confusion matrix, coefficient of variation, and User's accuracy and Producer's accuracy coefficients, the best algorithm for classifying land uses in the metropolitan area of Mashhad was determined in five classes: 1- Built-up areas, 2- Barren land, 3- Mountainous areas, 4- Green spaces, and 5- Water bodies. Results and Discussion: The results from the evaluation of standard deviation (SD) and coefficient of variation (CV) regarding the area share percentage in a LULC class by various algorithms indicate that barren lands were classified with higher accuracy, while water bodies and green spaces were classified with lower accuracy. The examination of U_Accuracy and P_Accuracy coefficients shows that the overall accuracy of the classification for all studied algorithms falls within the range of good to excellent. However, a more detailed examination of these algorithms reveals that the greatest challenge in class identification lies in built-up areas, mountainous regions, and green spaces, whereas the identification of barren lands faces fewer challenges. The Kappa coefficient and analyses based on the confusion matrix also demonstrate the variation in accuracy among each LULC classifier. The differences in the accuracy of the classifiers used are marginal, but these slight variations hold significant importance in the context of LULC planning. Given that these marginal differences are evident in sensitive land uses such as built-up areas and green spaces, selecting an algorithm with the highest accuracy and lowest error is of special importance.
Conclusion: The results of the Kappa coefficient evaluation and confusion matrix analyses indicate that the SVM approach has greater overall accuracy and a higher Kappa coefficient compared to RF and MLC methods. Specifically, the algorithms achieved overall accuracies of 0.93, 0.88, and 0.80, respectively. Therefore, Support Vector Machine demonstrates the highest accuracy and least error among the studied classifiers. Considering that numerous studies have shown that LULC mapping accuracy is related to time and location, it is suggested that future research analyse the accuracy of classifiers under different morphoclimatic and geomorphic conditions.
Keywords
Kaywords: Remote sensing
Land use classification
Support Vector Machine
Random Forest
Maximum Likelihood
Abburu, S. & Golla, S.B. 2015, Satellite Image Classification Methods and Techniques: A Review, International Journal of Computer Applications, 119(8).‏
Aburas, M.M., Ho, Y.M., Ramli, M.F. & Ash’aari, Z.H., 2016, The Simulation and Prediction of Spatio-Temporal Urban Growth Trends Using Cellular Automata Models: A Review, International Journal of Applied Earth Observation and Geoinformation, 52, PP. 380-389, https://doi.org/10.1016/j.jag.2016.07.007.
Aljanabi, F., Dedeoğlu, M. & Şeker, C., 2024, Environmental Monitoring of Land Use/Land Cover by Integrating Remote Sensing and Machine Learning Algorithms, Journal of Engineering and Sustainable Development, 28(4), PP. 455-466, https://doi.org/10.31272/jeasd.28.4.4.
Francisco Alonso-Sarría, F., Valdivieso-Ros, C., Francisco Gomariz-Castillo, F., 2024, Analysis of the hyperparameter optimisation of four machine learning satellite imagery classification methods, Computational Geosciences (2024) 28:551–571 https://doi.org/10.1007/s10596-024-10285-y.
 
Azizi Qalati, S., Rangzen, K., Taghizadeh, A. & Ahmadi, S., 2013, Modeling Land Use Changes Using Logistic Regression Method in LCM Model (Case Study: Kohmera Sorkhi Region of Fars Province), Iran Forest and Poplar Research, 22(4), PP. 585-596.
Baccari, N., Hamza, M.H., Slama, T., Sebei, A. & Rebai, N., 2024, Evaluation of SVM and RF Machine Learning Algorithms in Land Use/Land Cover Change Assessment: Tessa Watershed Case Study (Northwest of Tunisia)., Research Square, https://doi.org/10.21203/rs.3.rs-4359112/v1.
Bland, J.M. & Altman, D.G., 1995, Multiple Significance Tests: The Bonferroni Method, BMJ, 310(6973), P. 170, https://doi.org/10.1136/bmj.310.6973.170.
Chandra, M.A. & Bedi, S.S., 2021, Survey on SVM and their Application in Image Classification, International Journal of Information Technology, 13(5), PP. 1-11, https://doi.org/10.1007/s41870-017-0080-1.‏
Dahlke, J., Bogner, K., Mueller, M., Berger, T., Pyka, A. & Ebersberger, B., 2020, Is the Juice Worth the Squeeze? Machine Learning (ml) in and for Agent-Based Modelling (abm), arXiv Preprint arXiv, 2003.11985, https://doi.org/10.48550/arXiv.2003.11985.
de Espindola, G.M., da Costa Carneiro, E.L.N. & Façanha, A.C., 2017, Four Decades of Urban Sprawl and Population Growth in Teresina, Brazil, Applied Geography, 79, PP. 73-83, https://doi.org/10.1016/j.apgeog. 2016.12.018
Esmaili, A. & Ashjaei, H., 2019, Modeling Land Use Changes through Markov Chain and Using Geographic Information Systems and Remote Sensing (Case Study: Qom Province), Geography and Urban-Regional Studies 9(31), PP, 153-172, DOI: 10.22111/GAIJ.2019.4710.
Fleiss, J.L., 1971, Measuring Nominal Scale Agreement among Many Raters, Psychological Bulletin, 76(5), P. 378, https:// doi.org/10.1037/h0031619.
Gan, H., Sang, N., Huang, R., Tong, X. & Dan, Z., 2013, Using Clustering Analysis to Improve Semi-Supervised Classification, Neurocomputing, 101, PP. 290-298,‏ https:// doi.org/10.1016/j.neucom.2012.08.020.
Ganesh, B., Vincent, S., Pathan, S. & Benitez, S.R.G., 2023, Utilizing LANDSAT Data and the Maximum Likelihood Classifier for Analysing Land Use Patterns in Shimoga, Karnataka, Journal of Physics: Conference Series (Vol. 2571, No. 1, P. 012001), IOP Publishing, DOI: 10.1088/ 1742-6596/2571/1/012001.
Ghodsi, Z., Kheirkhah Zarkesh, M.M. & Gherzmecheshme, B., 2021, Comparison of Accuracy of Support Vector Machine and Random Forest Methods in Preparing Land Use Map and Crops, Using Sentinel-2 Multi-Temporal Images, Iranian Remote Sensing and GIS Journal, 12(4), PP. 73-92, DOI: 10.52547/GISJ.12.4.73.
Gholamali Fard, M., Jorabian Shoshtri, SH., Kohnouj, H. & Mirzaei, M., 2011, Modelling Land Use Changes of the Coasts of Mazandaran Province Using LCM in GIS Environment, Ecology, 38(64), PP. 109-124.
Glielmo, A., Husic, B.E., Rodriguez, A., Clementi, C., Noé, F. & Laio, A., 2021, Unsupervised Learning Methods for Molecular Simulation Data, Chemical Reviews, 121(16), PP. 9722-9758.‏ 10.1021/ acs.chemrev.0c01195.
Hosseini, M., Karimi, M., Saadi Masgari, M. & Heydari, M., 2016, Design and Implementation of an Integrated System of Urban Land Use Change Modeling, Geographical Sciences, 16(40), PP. 69-91.
Jahanbakhshi, F. & Ekhtesasi, M.R., 2019, Performance Evaluation of Three Image Classification Methods (Random Forest, Support Vector Machine and the Maximum Likelihood) in Land Use Mapping, Water and Soil Science, Science and Technology of Agriculture and Natural Resources, 22(4), PP. 235-247, SID, https://sid.ir/paper/394972/en.
Jasim, B.S., Jasim, O.Z. & AL-Hameedawi, A.N., 2024, Evaluating Land Use Land Cover Classification Based on Machine Learning Algorithms, Engineering and Technology Journal, 42(5), PP. 557-568, http://doi.org/10.30684/etj.2024.144585.1638.
Jiao, S.R., Song, J. & Liu, B., 2020, A Review of Decision Tree Classification Algorithms for Continuous Variables, Journal of Physics, Conference Series (Vol. 1651, No. 1, p. 012083), IOP Publishing, DOI: 10.1088/1742-6596/1651/1/012083.
Landis, J.R. & Koch, G.G., 1977, An Application of Hierarchical Kappa-Type Statistics in the Assessment of Majority Agreement Among Multiple Observers, Biometrics, 33(2), PP. 363-374, https:// doi.org/10.2307/2529786.
Louppe, G., 2014, Understanding Random Forests, Cornell University Library, 10.‏
Mahmoudzadeh, H., Derakhshani, K. & Momeni, S., 2019, Modeling the Effects of Marginalization on the Changes of Urmia City and Predicting the Physical Development of the City Using Satellite Images Until the Horizon of 1410, Human Geography Research, 51(4), PP. 90-871, 10.22059/JHGR.2018.230788.1007434.
Marudi, M., Ben-Gal, I. & Singer, G., 2024, A Decision Tree-Based Method for Ordinal Classification Problems, IISE Transactions, 56(9), PP. 960-974,‏ https:// doi.org/ 10.1080/24725854.2022.2081745.
Mashhad Municipality, 2016, Statistics of Mashhad City, Vice President of Planning and Human Capital Development of Mashhad Municipality with the Supervision of Statistics Management, Performance Analysis and Evaluation.
Mendoza, M.E., Granados, E.L., Geneletti, D., Pérez-Salicrup, D.R. & Salinas, V., 2011, Analysing Land Cover and Land Use Change Processes at Watershed Level: A Multitemporal Study in the Lake Cuitzeo Watershed, Mexico (1975–2003), Applied Geography, 31(1), PP. 237-250,‏ https:// doi.org/10.1016/j.apgeog.2010.05.010.
Mirakhorlou, M. & Rahimzadegan, S., 2018, Land use Change Modeling Using Markov-Cellular Automata Model and Multi-Criteria Decision Making in Talar Watershed, Scientific Journal of Mapping Sciences and Techniques, 8(1), PP. 85-99.
Mohammadi, J. & Mahmoud, A., 2012, Spatial Analysis and Urban Land Use Planning of Dogonbadan (Gachsaran), Geographical Research, 27(2), PP. 19-36.
Moradi, M. & Kasani, E., 2024, Optimal Classification Using Decision Tree, Mathematical Researches, 10(2), PP. 51-65.
Naswin, A. & Wibowo, A.P., 2023, Performance Analysis of the Decision Tree Classification Algorithm on the Pneumonia Dataset, International Journal of Artificial Intelligence in Medical Issues, 1(1), PP. 1-9.‏ 
Nti, I.K., Nyarko-Boateng, O., Adekoya, A.F., Weyori, B.A. & Adjei, H.P., 2023, Predicting Diabetes Using Cohen's Kappa Blending Ensemble Learning, International Journal of Electronic Healthcare, 13(1), PP. 57-70,‏ https://doi.org/ 10.1504/IJEH.2023. 128605.
Piryonesi, S.M., 2019, The Application of Data Analytics to Asset Management: Deterioration and Climate Change Adaptation in Ontario Roads, University of Toronto (Canada).‏
Pourkhabbaz, H., Mohammadyari, F., Aqdar, H. & Tavakoli, M., 2015, An Experimental Approach in Modeling Land Use Changes in Behbahan City Using Multi-Temporal Satellite Images, Town & Country Planning, 7(2), PP. 2008-7047, 10.22059/JTCP.2015.57187.
Qiang, W. & Zhongli, Z., 2011, Reinforcement Learning Model, Algorithms and Its Application, International Conference on Mechatronic Science, Electric Engineering and Computer (MEC) (PP. 1143-1146), IEEE 10.1109/MEC.2011.6025669.
Razafinimaro, A., Hajalalaina, A.R., Rakotonirainy, H. & Zafimarina, R., 2022, Land Cover Classification Based Optical Satellite Images Using Machine Learning Algorithms, International Journal of Advances in Intelligent Informatics, 8(3), PP. 362-380, https://doi.org/10.26555/ijain. v8i3.803.             
Ritsema van Eck, J. & Koomen, E., 2008, Characterising Urban Concentration and Land-Use Diversity in Simulations of Future Land Use, The Annals of Regional Science, 42, PP. 123-140,‏ https://doi.org/ 10.1007/s00168-007-0141-7.     
Sakai, T., Plessis, M.C., Niu, G. & Sugiyama, M., 2017, Semi-Supervised Classification Based on Classification from Positive and Unlabeled Data, International Conference on Machine Learning (PP. 2998-3006), PMLR.‏
Santarsiero, V., Nolè, G., Lanorte, A., Tucci, B., Cillis, G. & Murgante, B., 2022, Remote Sensing and Spatial Analysis for Land-Take Assessment in Basilicata Region (Southern Italy), Remote Sensing, 14(7), P. 1692, https://doi.org/10.3390/rs14071692.
Savitha, C. & Reshma, T., 2022, Performance Evaluation of Support Vector Machine and Random Forest Techniques for Land Use-Land Cover Classification—A Case Study on a Mili Scale Agricultural Watershed, Tadepalligudem, India, International Virtual Conference on Developments and Applications of Geomatics, PP. 379-392, Singapore: Springer Nature Singapore.‏ 
Setyanto, A. & Sunyoto, A., 2024, Comparative Study SVM and Random Forest Algorithms for the Classification of Terrestrial Visual Rock Types, IOP Conference Series: Earth and Environmental Science 1357(1), 10.1088/1755-1315/1357/1/ 012036.
Shi, S., Zhang, X. & Fan, W., 2019, Explaining the Predictions of any Image Classifier via Decision Trees, arXiv preprint arXiv, 1911.01058,
https://doi.org/10.48550/arXiv. 1911.01058.
Sowmya, D.R., Shenoy, P.D. & Venugopal, K.R., 2017, Remote Sensing Satellite Image Processing Techniques for Image Classification: A Comprehensive Survey, International Journal of Computer Applications, 161(11), PP. 24-37.‏ 
Talukdar, S., Singha, P., Mahato, S., Pal, S., Liou, Y.A. & Rahman, A., 2020, Land-Use Land-Cover Classification by Machine Learning Classifiers for Satellite Observations—A Review, Remote Sensing, 12(7), P. 1135,‏ https://doi.org/10.3390/rs12071135.    
Yimer, S.M., Bouanani, A., Kumar, N., Tischbein, B. & Borgemeister, C., 2024, Comparison of Different Machine-Learning Algorithms for Land Use Land Cover Mapping in a Heterogenous Landscape over the Eastern Nile River Basin, Ethiopia, Advances in Space Research, 74(5), PP. 2180-2199,‏ https://doi.org/10.1016/j.asr.2024.06.010.
Yin, M., Wortman Vaughan, J. & Wallach, H., 2019, Understanding the Effect of Accuracy on Trust in Machine Learning Models, Proceedings of the 2019 Chi Conference on Human Factors in Computing Systems (PP. 1-12),‏ https://doi.org/10.1145/ 3290605.3300509.