International Journal of Sensors, Wireless Communications and ControlInternational Journal of Sensors, Wireless Communications and ControlInternational Journal of Sensors, Wireless Communications and Control2210-32792210-3287Bentham Science64556410.2174/0122103279291431240216061325Computer and Information ScienceResearch ArticleSpectrum and Power Efficient Anti-jamming Approach for Cognitive Radio Networks Based on Reinforcement LearningJdeedHusseininfo@benthamscience.netAltabbanWissaminfo@benthamscience.netJamalSamerinfo@benthamscience.netDepartment of Telecommunication, Higher Institute for Applied Sciences and Technology0104202414427932811012025Copyright ©; 2024, Bentham Science Publishers2024Bentham Science Publishershttps://journals.eco-vector.com/2210-3279/article/view/645564

Background:Spectrum scarcity, spectrum efficiency, power constraints, and jamming attacks are core challenges that face wireless networks. While cognitive radio networks (CRNs) enable the sharing of licensed bands when they are unoccupied, the spectrum should be used efficiently by the secondary user (SU) to ensure a high data rate transmission. In addition, the mobility of the SUs makes power consumption a matter of concern in wireless networks. Because of the open environment, the jamming attack can easily deteriorate the performance and disrupt the connections.

Objectives:We aim to enhance the performance of CRN and establish more reliable connections for the SU in the presence of smart jammer by ensuring efficient spectrum utilization and extending the network lifetime.

Methods:To achieve our objectives, we propose an anti-jamming approach that adopts frequency hopping. Our approach assumes that SUs observe spectrum availability and channel gain. Then, SU learns the jammer behaviour and goes for the appropriate policy in terms of the number of data and control channels that optimize jointly spectrum efficiency and power consumption. Within, the interaction between the SU and the jammer is modelled as a zero-sum stochastic game, and we employ reinforcement learning (RL) to address this game.

Results:SUs learn the optimal policy that maximizes the spectrum efficiency and minimizes the power consumption in the presence of a smart jammer. Simulation results show that the low channel gain leads the SU to select a high number of data channels. However, when the channel gain is high, the SU increases the number of control channels to guarantee a more reliable connection. Taking into account the spectrum efficiency, SUs save their energy by decreasing the number of used channels. The proposed strategy achieves better performance in comparison with myopic learning and the random strategy.

Conclusion:Under a jamming attack, considering the gain of utilized channels, SUs select the appropriate number of control and data channels to ensure a reliable, efficient, and long-term connection.

Spectrum efficiencypower efficiencyanti-jammingcognitive radio networkreinforcement learningsecondary user.Akyildiz IF, Lee WY, Vuran MC, Mohanty S. NeXt generation/dynamic spectrum access/cognitive radio wireless networks: A survey. Comput Netw 2006; 50(13): 2127-59. doi: 10.1016/j.comnet.2006.05.001Mitola J. Cognitive radio an integrated agent architecture for software defined radio. Comput Sci Eng 2000.Cordeiro C, Challapali K, Birru D, Sai Shankar N. IEEE 802.22: the first worldwide wireless standard based on cognitive radios. First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks. 2005 DySPAN 2005, Baltimore, MD, USA 2005; pp. 328-37. doi: 10.1109/DYSPAN.2005.1542649Aref MA, Jayaweera SK, Yepez E. Survey on cognitive anti‐jamming communications. IET Commun 2020; 14(18): 3110-27. doi: 10.1049/iet-com.2020.0024Di Pietro R, Oligeri G. Jamming mitigation in cognitive radio networks. IEEE Netw 2013; 27(3): 10-5. doi: 10.1109/MNET.2013.6523802Amuru S, Tekin C, der Schaar M, Buehrer RM. Jamming bandits—a novel learning method for optimal jamming. IEEE Trans Wirel Commun 2016; 15(4): 2792-808. doi: 10.1109/TWC.2015.2510643Yang D, Xue G, Zhang J, Richa A, Fang X. Coping with a smart jammer in wireless networks: A stackelberg game approach. IEEE Trans Wirel Commun 2013; 12(8): 4038-47. doi: 10.1109/TWC.2013.071913121570Marti G. Kölle T, Studer C. Mitigating smart jammers in multi-user MIMO. IEEE Trans Signal Process 2023; 71: 756-71. doi: 10.1109/TSP.2023.3246226Slimeni F, Scheers B, Le Nir V, Chtourou Z, Attia R. Learning multi-channel power allocation against smart jammer in cognitive radio networks. 2016 International Conference on Military Communications and Information Systems (ICMCIS). Brussels, Belgium. 2016; pp. 1-7. doi: 10.1109/ICMCIS.2016.7496544Letafati M, Kuhestani A, Ng DWK, Behroozi H. A new frequency hopping-aided secure communication in the presence of an adversary jammer and an untrusted relay. 2020 IEEE International Conference on Communications Workshops, ICC Workshops 2020 - Proceedings. Dublin, Ireland, 2020; pp. 1-7. doi: 10.1109/ICCWorkshops49005.2020.9145441Chen KW, Chao CM, Lin CY, Yeh CC. Anti-jamming channel hopping protocol design based on channel occupancy probability for cognitive radio networks. Comput Netw 2022; 214: 109125. doi: 10.1016/j.comnet.2022.109125Quan H, Zhao H, Cui P. Anti-jamming frequency hopping system using multiple hopping patterns. Wirel Pers Commun 2015; 81(3): 1159-76. doi: 10.1007/s11277-014-2177-1Arjoune Y, Faruque S. Smart jamming attacks in 5G new radio: A review. 2020 10th Annual Computing and Communication Workshop and Conference. CCWC 2020, Las Vegas, NV, USA, 2020; pp. 1010-5. doi: 10.1109/CCWC47524.2020.9031175Liu Y, Ning P, Dai H, Liu A. Randomized differential DSSS: Jamming-resistant wireless broadcast communication. Proc IEEE INFOCOM 2010; 1-9. doi: 10.1109/INFCOM.2010.5462156Alagil A, Liu Y. Random allocation seed-DSSS broadcast communication against jamming attacks. In: Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering. LNICST 2019; 304: pp. 472-89. doi: 10.1007/978-3-030-37228-6_23Yan Q, Zeng H, Jiang T, Li M, Lou W, Hou YT. MIMO-based jamming resilient communication in wireless networks. Proc IEEE INFOCOM 2014; 2697-706. doi: 10.1109/INFOCOM.2014.6848218Akhlaghpasand H, Bjornson E, Razavizadeh SM. Jamming Suppression in Massive MIMO Systems. IEEE Trans Circuits Syst II Express Briefs 2020; 67(1): 182-6. doi: 10.1109/TCSII.2019.2902074Yan Q, Zeng H, Jiang T, Li M, Lou W, Hou YT. Jamming resilient communication using mimo interference cancellation. IEEE Trans Inf Forensics Security 2016; 11(7): 1486-99. doi: 10.1109/TIFS.2016.2535906Okyere B, Musavian L, Ozbek B, Busari SA, Gonzalez J. The resilience of massive MIMO PNC to jamming attacks in vehicular networks. IEEE Trans Intell Transp Syst 2021; 22(7): 4110-7. doi: 10.1109/TITS.2020.3016907Shen W, Ning P, He X, Dai H, Liu Y. MCR Decoding: A MIMO approach for defending against wireless jamming attacks. 2014 IEEE Conference on Communications and Network Security, CNS 2014. San Francisco, CA, USA. 2014; pp. 133-8. doi: 10.1109/CNS.2014.6997478Guosen Y, Xiaodong W, Madihian M. Design of anti-jamming coding for cognitive radio. GLOBECOM - IEEE Global Telecommunications Conference. Washington, DC, USA. 2007; pp. 4190-4. doi: 10.1109/GLOCOM.2007.797Yue G, Wang X. Anti-jamming coding techniques with application to cognitive radio. IEEE Trans Wirel Commun 2009; 8(12): 5996-6007. doi: 10.1109/TWC.2009.12.081627Pirayesh H, Zeng H. Jamming attacks and anti-jamming strategies in wireless networks: A comprehensive survey. IEEE Commun Surv Tutor 2022; 24(2): 767-809. doi: 10.1109/COMST.2022.3159185Noubir G, Lin G. Low-power DoS attacks in data wireless LANs and countermeasures. Mob Comput Commun Rev 2003; 7(3): 29-30. doi: 10.1145/961268.961277Lin G, Noubir G. On link layer denial of service in data wireless LANs. Wirel Commun Mob Comput 2005; 5(3): 273-84. doi: 10.1002/wcm.221Strasser M, Pöpper C, Čapkun S. Efficient uncoordinated fhss anti-jamming communication. Proceedings of the International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc). New Orleans, LA, USA, May 18-21. 2009; pp. 207-18. doi: 10.1145/1530748.1530778Shi Y, Lu X, An K, Li Y, Zheng G. Efficient index modulation based FHSS: A unified anti-jamming perspective. IEEE Internet Things J 2023; 11(2): 3458-72. doi: 10.1109/JIOT.2023.3296605Shi Y, An K, Lu X, Li Y. Enhanced index modulation-based frequency hopping: Resist power-correlated reactive jammer. IEEE Wirel Commun Lett 2022; 11(4): 751-5. doi: 10.1109/LWC.2022.3142253Xu J, Lou H, Zhang W, Sang G. An intelligent anti-jamming scheme for cognitive radio based on deep reinforcement learning. In: IEEE Access 2020; 8: 202563-72. doi: 10.1109/ACCESS.2020.3036027Krayani A, Alam AS, Marcenaro L, Nallanathan A, Regazzoni C. A novel resource allocation for anti-jamming in cognitive-UAVs: An active inference approach. IEEE Commun Lett 2022; 26(10): 2272-6. doi: 10.1109/LCOMM.2022.3190971Machuzak S, Jayaweera SK. Reinforcement learning based anti-jamming with wideband autonomous cognitive radios. 2016 IEEE/CIC International Conference on Communications in China (ICCC). Chengdu, China 2016; pp. 1-5. doi: 10.1109/ICCChina.2016.7636793Slimeni F, Chtourou Z, Ben Amor A. Reinforcement learning based anti-jamming cognitive radio channel selection. Proceedings of the International Conference on Advanced Systems and Emergent Technologies. IC_ASET, Hammamet, Tunisia 2020; pp. 431-5. doi: 10.1109/IC_ASET49463.2020.9318287Jiang W, Ren Y, Wang Y. Improving anti-jamming decision-making strategies for cognitive radar via multi-agent deep reinforcement learning. Digit Signal Process 2023; 135: 103952. doi: 10.1016/j.dsp.2023.103952Zhou Q, Li Y, Niu Y. Intelligent anti-jamming communication for wireless sensor networks: A multi-agent reinforcement learning approach. IEEE Open J Commun Soc 2021; 2: 775-84. doi: 10.1109/OJCOMS.2021.3056113Zhou W, Zhou Z, Niu Y, Zhou Q, Ding H. A fast anti-jamming algorithm based on imitation learning for WSN. Sensors 2023; 23: 9240. doi: 10.3390/s23229240Skokowski P, Kelner JM, Malon K, et al. Jamming and jamming mitigation for selected 5G military scenarios. Procedia Comput Sci 2022; 205: 258-67. doi: 10.1016/j.procs.2022.09.027Zhang Y, Jia L, Qi N, Xu Y, Wang M. Anti-jamming channel access in 5G ultra-dense networks: A game-theoretic learning approach. Digit Commun Netw 2023; 9(2): 523-33. doi: 10.1016/j.dcan.2022.04.031Lu X, Xiao L, Dai C, Dai H. UAV-aided cellular communications with deep reinforcement learning against jamming. IEEE Wirel Commun 2020; 27(4): 48-53. doi: 10.1109/MWC.001.1900207Krayani A, Baydoun M, Marcenaro L, Gao Y, Regazzoni CS. Smart jammer detection for self-aware cognitive UAV radios. IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, PIMRC. 31 August 2020 - 03 September 2020; London, UK. 1-7. doi: 10.1109/PIMRC48278.2020.9217331Wu Q, Wang H, Li X, Zhang B, Peng J. Reinforcement learning-based anti-jamming in networked UAV radar systems. Appl Sci 2019; 9: 5173. doi: 10.3390/app9235173Xiao L, Ding Y, Huang J, Liu S, Tang Y, Dai H. UAV anti-jamming video transmissions with QoE guarantee: A reinforcement learning-based approach. IEEE Trans Commun 2021; 69(9): 5933-47. doi: 10.1109/TCOMM.2021.3087787Khan AU, Abbas G, Abbas ZH, Waqas M, Hassan AK. Spectrum utilization efficiency in the cognitive radio enabled 5G-based IoT. J Netw Comput Appl 2020; 164: 102686. doi: 10.1016/j.jnca.2020.102686Khalaf OI, Ogudo KA, Singh M. A fuzzy-based optimization technique for the energy and spectrum efficiencies trade-off in cognitive radio-enabled 5G network. Symmetry 2021 2020; 13: 47. doi: 10.3390/sym13010047Chatterjee S, Maity SP, Acharya T. Energy-spectrum efficiency trade-off in energy harvesting cooperative cognitive radio networks. IEEE Trans Cogn Commun Netw 2019; 5(2): 295-303. doi: 10.1109/TCCN.2019.2903503Mughal DM, Shah ST, Chung MY. An efficient spectrum utilization scheme for energy-constrained IoT devices in cellular networks. IEEE Internet Things J 2021; 8(17): 13414-24. doi: 10.1109/JIOT.2021.3064330Jain P, Gupta A, Kumar N, Guizani M. Dynamic and efficient spectrum utilization for 6G with THz, mmWave, and RF band. IEEE Trans Vehicular Technol 2023; 72(3): 3264-73. doi: 10.1109/TVT.2022.3215487Paul A, Banerjee A, Maity SP. Throughput maximisation in cognitive radio networks with residual bandwidth. IET Commun 2019; 13(10): 1327-35. doi: 10.1049/iet-com.2018.5928Zheng K, Liu X, Zhu Y, Chi K, Liu K. Total throughput maximization of cooperative cognitive radio networks with energy harvesting. IEEE Trans Wirel Commun 2020; 19(1): 533-46. doi: 10.1109/TWC.2019.2946813Liu X, Xu B, Zheng K, Zheng H. Throughput maximization of wireless-powered communication network with mobile access points. IEEE Trans Wirel Commun 2023; 22(7): 4401-15. doi: 10.1109/TWC.2022.3225085Zheng K, Luo R, Wang Z, Liu X, Yao Y. Short-term and long-term throughput maximization in mobile wireless-powered internet of things. IEEE Internet Things J 2023. doi: 10.1109/JIOT.2023.3326440Chiaraviglio L, D’Andreagiovanni F, Liu W, et al. Multi-area throughput and energy optimization of UAV-aided cellular networks powered by solar panels and grid. IEEE Trans Mobile Comput 2021; 20(7): 2427-44. doi: 10.1109/TMC.2020.2980834Xie L, Xu J, Zeng Y. Common throughput maximization for UAV-enabled interference channel with wireless powered communications. IEEE Trans Commun 2020; 68(5): 3197-212. doi: 10.1109/TCOMM.2020.2971488Zheng K, Liu X, Wang B, Zheng H, Chi K, Yao Y. Throughput maximization of wireless-powered communication networks: An energy threshold approach. IEEE Trans Vehicular Technol 2021; 70(2): 1292-306. doi: 10.1109/TVT.2021.3050412Hu B, Wang L, Chen S, Cui J, Chen L. An uplink throughput optimization scheme for UAV-enabled urban emergency communications. IEEE Internet Things J 2022; 9(6): 4291-302. doi: 10.1109/JIOT.2021.3103892Zheng K, Jia X, Chi K, Liu X. DDPG-based joint time and energy management in ambient backscatter-assisted hybrid underlay CRNs. IEEE Trans Commun 2023; 71(1): 441-56. doi: 10.1109/TCOMM.2022.3221422Tian J, Xiao H, Sun Y, Hou D, Li X. Energy efficiency optimization-based resource allocation for underlay RF-CRN with residual energy and QoS guarantee. EURASIP J Wirel Commun Netw 2020; 2020(1): 216. doi: 10.1186/s13638-020-01824-zBabu TS, Rao SN, Satyanarayana P. A design of minimizing interference and maximizing throughput in cognitive radio network by joint optimization of the channel allocation and power control. Int J Wirel Inf Netw 2023; 30(2): 211-25. doi: 10.1007/s10776-023-00592-zNandan N, Majhi S, Wu HC. Beamforming and power optimization for physical layer security of mimo-noma based crn over imperfect csi. IEEE Trans Vehicular Technol 2021; 70(6): 5990-6001. doi: 10.1109/TVT.2021.3079136Aslani R, Rasti M. A distributed power control algorithm for energy efficiency maximization in wireless cellular networks. IEEE Wirel Commun Lett 2020; 9(11): 1975-9. doi: 10.1109/LWC.2020.3010156Erpek T, Sagduyu YE, Shi Y. Deep learning for launching and mitigating wireless jamming attacks. IEEE Trans Cogn Commun Netw 2019; 5(1): 2-14. doi: 10.1109/TCCN.2018.2884910Gwon Y, Dastangoo S, Fossa C, Kung HT. Competing mobile network game: Embracing antijamming and jamming strategies with reinforcement learning. 2013 IEEE Conference on Communications and Network Security, CNS, National Harbor. MD, USA, 2013; pp. 28-36. doi: 10.1109/CNS.2013.6682689Ibrahim K, Ng SX, Qureshi IM, Malik AN, Muhaidat S. Anti-jamming game to combat intelligent jamming for cognitive radio networks. IEEE Access 2021; 9: 137941-56. doi: 10.1109/ACCESS.2021.3117563Ma Y, Liu K, Luo X. Game theory based multi-agent cooperative anti-jamming for mobile ad hoc networks. 2022 IEEE 8th International Conference on Computer and Communications (ICCC). Chengdu, China, 2022; pp. 901-5. doi: 10.1109/ICCC56324.2022.10065839Hanawal MK, Abdel-Rahman MJ, Krunz M. Game theoretic antijamming dynamic frequency hopping and rate adaptation in wireless systems. 2014 12th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt). Hammamet, Tunisia 2014; pp. 247-54. doi: 10.1109/WIOPT.2014.6850306Gouissem A, Abualsaud K, Yaacoub E, Khattab T, Guizani M. IoT anti-jamming strategy using game theory and neural network 2020 International Wireless Communications and Mobile Computing. IWCMC Limassol, Cyprus 2020; pp. 770-6. doi: 10.1109/IWCMC48107.2020.9148376Jia L, Qi N, Chu F, et al. Game-theoretic learning anti-jamming approaches in wireless networks. IEEE Commun Mag 2022; 60(5): 60-6. doi: 10.1109/MCOM.001.00496Noori H, Sadeghi Vilni S. Jamming and anti-jamming in interference channels: A stochastic game approach. IET Commun 2020; 14(4): 682-92. doi: 10.1049/iet-com.2019.0637Van Huynh N, Nguyen DN, Hoang DT, Dutkiewicz E. "Jam Me If You Can:" Defeating jammer with deep dueling neural network architecture and ambient backscattering augmented communications. IEEE J Sel Areas Comm 2019; 37(11): 2603-20. doi: 10.1109/JSAC.2019.2933889Han G, Xiao L, Poor HV. Two-dimensional anti-jamming communication based on deep reinforcement learning. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans, LA, USA. 2017; pp. 2087-91. doi: 10.1109/ICASSP.2017.7952524Wu Y, Wang B, Liu KJR, Clancy TC. Anti-jamming games in multi-channel cognitive radio networks. IEEE J Sel Areas Comm 2012; 30(1): 4-15. doi: 10.1109/JSAC.2012.120102Slimeni F, Scheers B, Chtourou Z, Le Nir V. Jamming mitigation in cognitive radio networks using a modified Q-learning algorithm. 2015 International Conference on Military Communications and Information Systems (ICMCIS). Cracow, Poland. 2015; pp. 1-7. doi: 10.1109/ICMCIS.2015.7158697Xiao L, Li Y, Liu J, Zhao Y. Power control with reinforcement learning in cooperative cognitive radio networks against jamming. J Supercomput 2015; 71(9): 3237-57. doi: 10.1007/s11227-015-1420-1Wang B, Yongle Wu, Liu KJR, Clancy TC. An anti-jamming stochastic game for cognitive radio networks. IEEE J Sel Areas Comm 2011; 29(4): 877-89. doi: 10.1109/JSAC.2011.110418Xiao L, Xie C, Min M, Zhuang W. User-centric view of unmanned aerial vehicle transmission against smart attacks. IEEE Trans Vehicular Technol 2018; 67(4): 3420-30. doi: 10.1109/TVT.2017.2785414Sharma H, Kumar N, Tekchandani R. Mitigating jamming attack in 5g heterogeneous networks: A federated deep reinforcement learning approach. IEEE Trans Vehicular Technol 2023; 72(2): 2439-52. doi: 10.1109/TVT.2022.3212966Yao F, Jia L. A collaborative multi-agent reinforcement learning anti-jamming algorithm in wireless networks. IEEE Wirel Commun Lett 2019; 8(4): 1024-7. doi: 10.1109/LWC.2019.2904486Yang H, Xiong Z, Zhao J, et al. Intelligent reflecting surface assisted anti-jamming communications: A fast reinforcement learning approach. IEEE Trans Wirel Commun 2021; 20(3): 1963-74. doi: 10.1109/TWC.2020.3037767Chen M, Liu W, Zhang N, et al. GPDS: A multi-agent deep reinforcement learning game for anti-jamming secure computing in MEC network. Expert Syst Appl 2022; 210: 118394. doi: 10.1016/j.eswa.2022.118394Pourranjbar A, Kaddoum G, Ferdowsi A, Saad W. Reinforcement learning for deceiving reactive jammers in wireless networks. IEEE Trans Commun 2021; 69(6): 3682-97. doi: 10.1109/TCOMM.2021.3062854Liu X, Xu Y, Jia L, Wu Q, Anpalagan A. Anti-jamming communications using spectrum waterfall: A deep reinforcement learning approach. IEEE Commun Lett 2018; 22(5): 998-1001. doi: 10.1109/LCOMM.2018.2815018Bi Y, Wu Y, Hua C. Deep reinforcement learning based multi-user anti-jamming strategy. ICC 2019 - 2019 IEEE International Conference on Communications (ICC). Shanghai, China 2019; pp. 1-6. doi: 10.1109/ICC.2019.8761848Littman ML. Markov games as a framework for multi-agent reinforcement learning. roceedings of the Eleventh International Conference. Rutgers University. New Brunswick, NJ, July 10-13, 1994; pp. 157-63. doi: 10.1016/B978-1-55860-335-6.50027-1Singh RS, Prasad A, Moven RM, Deva Sarma HK. Denial of service attack in wireless data network: A survey. 2017 Devices for Integrated Circuit (DevIC). Kalyani, India, 2017; pp. 354-9. doi: 10.1109/DEVIC.2017.8073968Chan A, Liu X, Noubir G, Thapa B. Broadcast control channel jamming: Resilience and identification of traitors. 2007 IEEE International Symposium on Information Theory. Nice, France. 2007; pp. 2496-500. doi: 10.1109/ISIT.2007.4557594Shapley LS. Stochastic games. Proc Natl Acad Sci USA 1953; 39(10): 1095-100. doi: 10.1073/pnas.39.10.1095 PMID: 16589380Solan E, Vieille N. Stochastic games. Proc Natl Acad Sci USA 2015; 112(45): 13743-6. doi: 10.1073/pnas.1513508112 PMID: 26556883Cadeau W, Li X, Xiong C. Markov model based jamming and anti-jamming performance analysis for cognitive radio networks. Commun Netw 2014; 6(2): 76-85. doi: 10.4236/cn.2014.62010Qinqing Zhang, Kassam SA. Finite-state markov model for rayleigh fading channels. IEEE Trans Commun 1999; 47(11): 1688-92. doi: 10.1109/26.803503