(Cambridge Institute of Technology, Bengaluru, Visvesvaraya Technological University, Belagavi - 590018, India, India)(2) Shashikumar Dandinashivara Revanna
(Sai Vidya Institute of Technology, Bengaluru, Visvesvaraya Technological University, Belagavi - 590018, India, India)*corresponding author
AbstractEfficient resource scheduling in Hadoop remains a challenging problem due to the presence of diverse workloads and varying priority requirements in cluster environments. Traditional YARN schedulers such as FIFO, Fair, and Capacity often struggle to simultaneously balance data locality, responsiveness, fairness, and priority handling, which can lead to increased waiting times and inefficient resource utilization. To address these limitations, this study proposes a Quantum-Inspired Priority Scheduler (QIPS) that integrates a Quantum-behaved Particle Swarm Optimization (QPSO) mechanism within the YARN Resource Manager to enhance job–node assignment decisions. The proposed scheduler considers multiple performance criteria, including latency, resource utilization, data locality, and priority awareness, enabling more adaptive and balanced scheduling under dynamic workload conditions. A hybrid implementation combining Java and Python is developed, where YARN handles job execution while the QPSO module performs optimization. Experimental evaluation on a multi-node Hadoop 3.3.4 cluster using synthetic workloads shows that QIPS effectively reduces deadline penalties and improves data locality, while maintaining competitive performance across other scheduling metrics. These findings indicate that quantum-inspired optimization offers a promising direction for achieving efficient and balanced resource scheduling in distributed systems.
KeywordsData Locality; Distributed Scheduling; Hadoop YARN; Priority-aware Optimization; Resource Management
|
DOIhttps://doi.org/10.26555/ijain.v12i3.2302 |
Article metricsAbstract views : 228 | PDF views : 15 |
Cite |
Full Text Download
|
References
[1] S. Hedayati, N. Maleki, T. Olsson, F. Ahlgren, M. Seyednezhad, and K. Berahmand, “MapReduce scheduling algorithms in Hadoop: a systematic study,” J. Cloud Comput., vol. 12, no. 1, p. 143, 2023, doi: 10.1186/s13677-023-00520-9.
[2] M. Hussain, L.-F. Wei, A. Rehman, M. Ali, S. M. Waqas, and F. Abbas, “Cost-aware quantum-inspired genetic algorithm for workflow scheduling in hybrid clouds,” J. Parallel Distrib. Comput., vol. 191, p. 104920, 2024, doi: 10.1016/j.jpdc.2024.104920.
[3] S. K. Sood, “Quantum-inspired metaheuristic algorithms for Industry 4.0: A scientometric analysis,” Eng. Appl. Artif. Intell., vol. 139, p. 109635, 2025, doi: 10.1016/j.engappai.2024.109635.
[4] R. Ghazali, S. Adabi, A. Rezaee, D. G. Down, and A. Movaghar, “CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning,” J. Cloud Comput., vol. 11, no. 1, p. 45, 2022, doi: 10.1186/s13677-022-00322-5.
[5] A. M. Rahmani, E. Y. Chamzini, M. Pourshaban, and M. Hosseinzadeh, “Scheduling of big data workflows in the Hadoop framework with heterogeneous computing cluster,” Arab. J. Sci. Eng., vol. 50, no. 15, pp. 12449–12461, 2025, doi: 10.1007/s13369-024-09779-9.
[6] Z. Zhang, C. Xu, S. Xu, L. Huang, and J. Zhang, “Towards optimized scheduling and allocation of heterogeneous resource via graph-enhanced EPSO algorithm,” J. Cloud Comput., vol. 13, no. 1, p. 108, 2024, doi: 10.1186/s13677-024-00670-4.
[7] B. Qureshi, “Optimizing Hadoop Scheduling in Single-Board-Computer-Based Heterogeneous Clusters,” Computation, vol. 12, no. 5, p. 96, 2024, doi: 10.3390/computation12050096.
[8] U. K. Lilhore et al., “QHRMOF: A Quantum-Inspired hybrid Multi-Objective framework for Energy-Efficient task scheduling and load balancing in cloud computing,” J. Cloud Comput., vol. 14, no. 1, p. 54, 2025, doi: 10.1186/s13677-025-00777-2.
[9] M. Cui and Y. Wang, “An effective QoS-aware hybrid optimization approach for workflow scheduling in cloud computing,” Sensors, vol. 25, no. 15, p. 4705, 2025, doi: 10.3390/s25154705.
[10] K. Fu, J. Liu, M. Chen, and H. Zhang, “Solving flexible job-shop scheduling problems based on quantum computing,” Entropy, vol. 27, no. 2, p. 189, 2025, doi: 10.3390/e27020189.
[11] S. P. Singh, G. Kumar, U. Ahirwar, S. Selvarajan, and F. Khan, “Multi-objective quantum hybrid evolutionary algorithms for enhancing quality-of-service in internet of things,” Sci. Rep., vol. 15, no. 1, p. 14795, 2025, doi: 10.1038/s41598-025-99429-3.
[12] Y. Bai, Y. Sui, X. Deng, and X. Wang, “Quantum-inspired robust optimization for coordinated scheduling of PV-hydrogen microgrids under multi-dimensional uncertainties,” Sci. Rep., vol. 15, no. 1, p. 29589, 2025, doi: 10.1038/s41598-025-12280-4.
[13] H. Tang and J. Dong, “Solving flexible job-shop scheduling problem with heterogeneous graph neural network based on relation and deep reinforcement learning,” Machines, vol. 12, no. 8, p. 584, 2024, doi: 10.3390/machines12080584.
[14] Z. Du, J. Wang, and H. Li, “Quantum firefly algorithm: a novel approach for quantum circuit scheduling optimization,” Electronics, vol. 14, no. 11, p. 2123, 2025, doi: 10.3390/electronics14112123.
[15] D. Fitzek, T. Ghandriz, L. Laine, M. Granath, and A. F. Kockum, “Applying quantum approximate optimization to the heterogeneous vehicle routing problem,” Sci. Rep., vol. 14, no. 1, p. 25415, 2024, doi: 10.1038/s41598-024-76967-w.
[16] F. Shabestari, A. M. Rahmani, N. J. Navimipour, and S. Jabbehdari, “A YARN-based Energy-Aware Scheduling Method for Big Data Applications under Deadline Constraints: F. Shabestari et al.,” J. Grid Comput., vol. 20, no. 4, p. 38, 2022, doi: 10.1007/s10723-022-09627-w.
[17] M. Bergui, S. Hourri, S. Najah, and N. S. Nikolov, “Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques,” J. Big Data, vol. 11, no. 1, p. 98, 2024, doi: 10.1186/s40537-024-00964-z.
[18] B. Qureshi, “Adaptive Multi-Criteria Selection for Efficient Resource Allocation in Frugal Heterogeneous Hadoop Clusters,” Electronics, vol. 13, no. 10, p. 1836, 2024, doi: 10.3390/electronics13101836.
[19] Y. Li, W. Song, B. Jin, X. Zuo, Y. Li, and K. Chen, “A SqueeSAR Spatially Adaptive Filtering Algorithm Based on Hadoop Distributed Cluster Environment,” Appl. Sci., vol. 13, no. 3, p. 1869, 2023, doi: 10.3390/app13031869.
[20] J. Zhou, B. Liu, and J. Gao, “A task scheduling algorithm with deadline constraints for distributed clouds in smart cities,” PeerJ Comput. Sci., vol. 9, p. e1346, 2023, doi: 10.7717/peerj-cs.1346.
[21] J. Wang, S. Li, X. Zhang, F. Wu, and C. Xie, “Deep reinforcement learning task scheduling method based on server real-time performance,” PeerJ Comput. Sci., vol. 10, p. e2120, 2024, doi: 10.7717/peerj-cs.2120.
[22] Y. Sang, J. Cheng, B. Wang, and M. Chen, “A three-stage heuristic task scheduling for optimizing the service level agreement satisfaction in device-edge-cloud cooperative computing,” PeerJ Comput. Sci., vol. 8, p. e851, 2022, doi: 10.7717/peerj-cs.851.
[23] D. Alsadie, “Advancements in heuristic task scheduling for IoT applications in fog-cloud computing: challenges and prospects,” PeerJ Comput. Sci., vol. 10, p. e2128, 2024, doi: 10.7717/peerj-cs.2128.
[24] P. Thanapol, K. Lavangnananda, F. Leprévost, A. Glad, J. Schleich, and P. Bouvry, “Round-based mechanism and job packing with model-similarity-based policy for Scheduling DL Training in GPU Cluster,” Appl. Sci., vol. 14, no. 6, p. 2349, 2024, doi: 10.3390/app14062349.
[25] V. Thesma, G. C. Rains, and J. Mohammadpour Velni, “Development of a low-cost distributed computing pipeline for high-throughput cotton phenotyping,” Sensors, vol. 24, no. 3, p. 970, 2024, doi: 10.3390/s24030970.
[26] S. N. Hegde, D. B. Srinivas, M. A. Rajan, S. Rani, A. Kataria, and H. Min, “Multi-objective and multi constrained task scheduling framework for computational grids,” Sci. Rep., vol. 14, no. 1, p. 6521, 2024, doi: 10.1038/s41598-024-56957-8.
[27] V. P. Verma, S. Kumar, S. Kumar, N. S. Naik, and R. Dubey, “Optimizing Spark job scheduling with distributional deep learning in cloud environments,” J. Cloud Comput., vol. 14, no. 1, p. 59, 2025, doi: 10.1186/s13677-025-00773-6.
[28] A. C. Ikegwu, H. F. Nweke, E. Mkpojiogu, C. V. Anikwe, S. A. Igwe, and U. R. Alo, “Recently emerging trends in big data analytic methods for modeling and combating climate change effects,” Energy Informatics, vol. 7, no. 1, pp. 1–28, 2024, doi: 10.1186/s42162-024-00307-5.
[29] S. Vengadeswaran, S. R. Balasundaram, and P. Dhavakumar, “IDaPS Improved data-locality aware data placement strategy based on Markov clustering to enhance MapReduce performance on Hadoop,” J. King Saud Univ. Comput. Inf. Sci., vol. 36, no. 3, p. 101973, 2024, doi: 10.1016/j.jksuci.2024.101973.
[30] M. Thiyyakat, S. Kalambur, and D. Sitaram, “Constraint-Aware Federated Scheduling for Data Center Workloads,” IoT, vol. 4, no. 4, pp. 534–557, 2023, doi: 10.3390/iot4040023.
[31] N. Du, C. Wu, A. Hou, W. Nie, and R. Song, “Efficient Scheduling for GPU-Based Neural Network Training via Hybrid Reinforcement Learning and Metaheuristic Optimization,” Big Data Cogn. Comput., vol. 9, no. 11, p. 284, 2025, doi: 10.3390/bdcc9110284.
[32] H. Wang, E. Deng, J. Li, and C. Zhang, “Edge computing resource scheduling method based on container elastic scaling,” PeerJ Comput. Sci., vol. 10, p. e2379, 2024, doi: 10.7717/peerj-cs.2379.
[33] Y. Teng and Z. Liu, “Edge computing task scheduling method based on user’s social relations: a construction and solution for Smart City Library,” PeerJ Comput. Sci., vol. 10, p. e2457, 2024, doi: 10.7717/peerj-cs.2457.
[34] A. Muhammad and M. A. Qadir, “MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput,” PeerJ Comput. Sci., vol. 8, p. e1077, 2022, doi: 10.7717/peerj-cs.1077.
[35] M. Elhmadany, I. Elmadah, and H. E. Abdelmunim, “Instance segmentation on distributed deep learning big data cluster,” J. Big Data, vol. 11, no. 1, p. 6, 2024, doi: 10.1186/s40537-023-00871-9.
[36] L. Chourasiya et al., “Advanced system log analyzer for anomaly detection and cyber forensic investigations using LSTM and transformer networks,” J. Cloud Comput., vol. 14, no. 1, p. 60, 2025, doi: 10.1186/s13677-025-00789-y.
[37] Z. Wu, X. Lv, Y. Yun, and W. Duan, “A parallel sequential SBAS processing framework based on Hadoop distributed computing,” Remote Sens., vol. 16, no. 3, p. 466, 2024, doi: 10.3390/rs16030466.
[38] D. Rahbari, “Analyzing meta-heuristic algorithms for task scheduling in a fog-based IoT application,” Algorithms, vol. 15, no. 11, p. 397, 2022, doi: 10.3390/a15110397.
[39] M. N. Aktan and H. Bulut, “Metaheuristic task scheduling algorithms for cloud computing environments,” Concurr. Comput. Pract. Exp., vol. 34, no. 9, p. e6513, 2022, doi: 10.1002/cpe.6513.
[40] C. Wang, Z. Wang, S. Zhang, X. Liu, and J. Tan, “Reinforced quantum-behaved particle swarm-optimized neural network for cross-sectional distortion prediction of novel variable-diameter-die-formed metal bent tubes,” J. Comput. Des. Eng., vol. 10, no. 3, pp. 1060–1079, 2023, doi: 10.1093/jcde/qwad037.
[41] B. Liu, Y. Zhou, Q. Luo, and H. Huang, “Quantum-inspired African vultures optimization algorithm with elite mutation strategy for production scheduling problems,” J. Comput. Des. Eng., vol. 10, no. 4, pp. 1767–1789, 2023, doi: 10.1093/jcde/qwad078.
[42] A. Dong and S.-K. Lee, “The study of an improved particle swarm optimization algorithm applied to economic dispatch in microgrids,” Electronics, vol. 13, no. 20, p. 4086, 2024, doi: 10.3390/electronics13204086.
[43] S. Lim and D. Park, “Improving Hadoop MapReduce performance on heterogeneous single board computer clusters,” Futur. Gener. Comput. Syst., vol. 160, pp. 752–766, 2024, doi: 10.1016/j.future.2024.06.025.
[44] H. Hou, S. N. A. Jawaddi, and A. Ismail, “Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review,” Futur. Gener. Comput. Syst., vol. 151, pp. 214–231, 2024, doi: 10.1016/j.future.2023.10.002.
[45] M. Hosseinzadeh, M. Y. Ghafour, H. K. Hama, B. Vo, and A. Khoshnevis, “Multi-Objective Task and Workflow Scheduling Approaches in Cloud Computing: a Comprehensive Review: Multi-objective Task and Workflow Scheduling Approaches in Cloud Computing: A Comprehensive Review,” J. Grid Comput., vol. 18, no. 3, pp. 327–356, 2020, doi: 10.1007/s10723-020-09533-z.
[46] S. A. Murad, A. J. M. Muzahid, Z. R. M. Azmi, M. I. Hoque, and M. Kowsher, “A review on job scheduling technique in cloud computing and priority rule based intelligent framework,” J. King Saud Univ. Inf. Sci., vol. 34, no. 6, pp. 2309–2331, 2022, doi: 10.1016/j.jksuci.2022.03.027.
[47] G. Kotikam and L. Selvaraj, “YARN Schedulers for Hadoop MapReduce Jobs: Design Goals, Issues and Taxonomy,” Recent Adv. Comput. Sci. Commun. (Formerly Recent Patents Comput. Sci., vol. 16, no. 6, pp. 44–55, 2023, doi: 10.2174/2666255816666220831125012.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0