AI-DRIVEN FRAMEWORK FOR SCALABLE, SECURE, AND INTELLIGENT BIG DATA MANAGEMENT IN CLOUD ENVIRONMENTS
DOI:
https://doi.org/10.71146/kjmr949Keywords:
Big Data Management, Artificial Intelligence, Cloud Computing, Machine Learning, Data Security, Anomaly Detection, Scalability, Resource OptimizationAbstract
Research is actively conducted on the significance of big data management because of the growth in the amount of data created through cloud technologies, Internet of Things (IoT), company databases, social networks, and security technologies. The existing approach of using batch processing and policies does not have the ability to satisfy all four requirements for scalability, performance, privacy protection, and flexibility at once. This paper considers the design of an AI-driven big data management system that uses cloud computing, distributed computing, machine learning classification, outlier detection, resource prediction, and control. The systematic literature review was performed on the papers published between 2021 and 2026, while the artificial benchmark dataset comprised of 50 test scenarios was built to compare the performance of the proposed architecture and traditional cloud architecture. According to the simulation outcomes, the proposed framework provides an average reduction in latency by 35.25%, increase in throughput by 27.99%, decrease in costs by 23.29%, and improvement in security/anomaly detection metrics by 9.13 percentage points. This research offers a unified classification of problems related to big data management, an operational design of AI-cloud architecture, and a reusable benchmark model for future verification in real-world corporate applications.
Downloads
References
1. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008.
2. M. Zaharia et al., "Apache Spark: A Unified Engine for Big Data Processing," Communications of the ACM, vol. 59, no. 11, pp. 56-65, 2016.
3. X. Wu et al., "Data Mining with Big Data," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 1, pp. 97-107, 2014.
4. R. Buyya, C. Vecchiola, and S. T. Selvi, Mastering Cloud Computing: Foundations and Applications Programming. Morgan Kaufmann, 2013.
5. NIST, "The NIST Definition of Cloud Computing," Special Publication 800-145, 2011.
6. D. Laney, "3D Data Management: Controlling Data Volume, Velocity, and Variety," META Group, 2001.
A. Labrinidis and H. V. Jagadish, "Challenges and Opportunities with Big Data," Proceedings of the VLDB Endowment, vol. 5, no. 12, pp. 2032-2033, 2012.
7. S. Sagiroglu and D. Sinanc, "Big Data: A Review," International Conference on Collaboration Technologies and Systems,
8. pp. 42-47, 2013.
9. C. Chen, S. Mao, and Y. Liu, "Big Data: A Survey," Mobile Networks and Applications, vol. 19, pp. 171-209, 2014.
10. N. Al-Mekhlal and B. M. Alabsi, "A Survey of Big Data Management: Taxonomy and Challenges," Journal of Big Data, 2023.
A. Al-Mazrawe and A. K. Al-Mousa, "Anomaly Detection in Cloud Network: A Review," BIO Web of Conferences, 2024.
11. S. Baimukhanov et al., "Enhancing ML-based Anomaly Detection in Data-Intensive IoT-Edge-Cloud Ecosystems," Expert Systems with Applications, 2025.
12. IBM Research, "Anomaly Detection in Large-Scale Cloud Systems," arXiv preprint arXiv:2411.09047, 2024.
13. M. Armbrust et al., "Above the Clouds: A Berkeley View of Cloud Computing," University of California, Berkeley, 2009.
A. Gandomi and M. Haider, "Beyond the Hype: Big Data Concepts, Methods, and Analytics," International Journal of Information Management, vol. 35, no. 2, pp. 137-144, 2015.
14. K. Kambatla, G. Kollias, V. Kumar, and A. Grama, "Trends in Big Data Analytics," Journal of Parallel and Distributed Computing, vol. 74, no. 7, pp. 2561-2573, 2014.
15. J. Manyika et al., "Big Data: The Next Frontier for Innovation, Competition, and Productivity," McKinsey Global Institute, 2011.
16. L. Wang et al., "Cloud-Based Big Data Systems," IEEE, 2015.
17. P. Mell and T. Grance, "The NIST Definition of Cloud Computing," NIST, 2011.
18. Y. Li et al., "Scalable Big Data Systems: A Survey," IEEE Access, 2020.
19. Z. Khan et al., "Future Challenges of Big Data," IEEE Access, 2021.
A. Jain et al., "Security and Privacy Issues in Big Data," IEEE, 2016.
20. P. Russom, "Big Data Analytics," TDWI Best Practices Report, 2011.
21. N. Kshetri, "Big Data’s Impact on Privacy, Security and Consumer Welfare," Telecommunications Policy, 2014.
22. M. A. Ferrag et al., "Deep Learning for Cyber Security Intrusion Detection: Approaches, Datasets, and Comparative Study," Journal of Information Security and Applications, 2020.
23. R. Ranjan, "Streaming Big Data Processing in Datacenter Clouds," IEEE Cloud Computing, 2014.
24. F. Li et al., "A Survey on Edge Computing for the Internet of Things," IEEE Access, 2018.
A. Nazir et al., "Machine Learning-Based Cloud Data Classification and Intrusion Detection," 2024.
25. Google Cloud, "What is Artificial Intelligence?" Google Cloud Learn, accessed 2026.
26. IBM, "What is Artificial Intelligence?" IBM Think, accessed 2026.
Downloads
Published
Issue
Section
Categories
License
Copyright (c) 2026 Muhammad Ahsan Hayat, Hamna Anis, Maryam Shaikh (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
KJMR publishes all articles as open access under CC BY 4.0, allowing anyone to share and adapt the work, even commercially, with proper credit, a license link, and clear notice of changes, without implying endorsement. Authors retain copyright while granting the journal non-exclusive publishing and archiving rights and may self-archive without embargo. Third-party material requires proper permission, and the journal ensures long-term free access through its website and archiving partners.
How to Cite
