Training Simulation for Cyber Security Novice Analysts

Training Simulation for Cyber Security Novice Analysts based on Cognitive Analysis of Cyber Security Experts

Abstract

In the world of digitization cyber security is becoming a greater concern for today’s society, with attacks on systems now being a more frequent and complex than ever. It makes it extremely hard for a trainee cyber security analyst to acquire the expert level skill set in the domain. This situation leads to have a need for a better training of cyber defense analysts. Major part of the job for a cyber security analyst to identify the false alarms correctly. This paper presents a cognitive task analysis approach for addressing this need for better training model focused on false alarm detection. The primary objective is to capture and characterize the performance of a cyber security expert to tackle the complex threat and incorporate it in the training model in order to provide effective training for the cyber situation awareness. To make it extremely effective it is crucial to design realistic training scenarios. As a part of the utilization of cognitive task analysis technique this paper focuses mainly on the improved training model for accurate identification of false alarms, helping a trainee to during performance to think and act as experts. To tackle the challenge of overloading information faced by cyber analysts, it proposes an attack-specific checklist items. During training, cyber analysts can adjust their own checklist items and set thresholds so that cyber attacks can be detected more quickly. Since the time required for cyber analysts to recognize, analyze and identify a threat as a false alarm is critical, we evaluate the performance of cyber analysts against the ideal timeline based on their response time.

Keywords: Cyber Attacks, Situation Awareness, Training for Cyber Security Experts

Training Simulation for Cyber Security Professionals

Cyber security is a large-scale societal problem. The threat to organizations and governments has continued to grow as we become increasingly dependent on information technology; meanwhile, the entities behind cyber attacks grow in sophistication. Low and slow attacks, also called advanced persistent threats, are a new category of cyber security threat designed to exist undetected over an extended period of time and disrupt the processes of an organization. In response, the role of the cyber security professional has developed as a specialized subset within information technology careers. Cyber security professionals are individuals who are responsible for ensuring the ongoing security of their organization’s computer network. Recent high-profile cases of network intrusions underscore the vulnerabilities in current information technology in banking, healthcare, retail, and in the government.

In general, cyber security professionals “protect, monitor, analyze, detect and respond to unauthorized activity,” a task called computer network defense (CND). Because of the large and growing volume of network activity, unaided performance of this task is impossible in large organizations. To reduce the human information processing requirements, automated tools are used. One example is an intrusion detection system (IDS), which examines server log files to find patterns associated with anomalies. When such a pattern is found, cyber security professionals can be alerted to investigate. However, IDSs are limited in their sophistication and reliability; this has been true of most forms of automation for CND. Because of this, CND is a joint human-machine collaborative task in which people depend on automated tools to perform their jobs but must remain “in the loop” as an information processor and decision maker.           Consequently, the cyber security professional is a critical line of defense in CND. Effective human decision making is a determinant of successful cyber security. Hence there is a need for training of cyber security analysts. It has been established that situation awareness (SA), a cognitive state resulting from a process of situation assessment, is a predictor of human performance across domains, and research has established its role in CND, where it is called cyber SA. In other words, cyber SA, as goal-relevant knowledge held during task performance, predicts threat response by describing whether cyber security professionals have adequate awareness of relevant elements in the task environment.

In cyber situational awareness, cyber analysts have to collect data and seek cues that form attack tracks, find the impact of attack tracks, and anticipate moves (actions, targets, time) of attackers. Due to the enormous size and complexity of network, cyber analysts face extraordinary cognitive challenges. First, the environment from which a cyber analyst has to perceive salient cues is vastly larger and more difficult to comprehend. Second, the speed at which the cyberspace changes is much faster, where new offensive technologies are constantly being developed. Third, the cyber analyst only sees the information that his/her (software) sensors are capable of detecting in a form that can be rendered on monitor screen. Furthermore, cyber analysts are given with large amounts of information (such as various IDS and audit logs) to look through, and CSA demands that various pieces of information be connected in both space and time. This connection necessitates team collaboration among cyber analysts working at different levels and on different parts of the system. As cyber attacks are becoming more frequent and more complex, the need for more effective training of cyber analysts and their collaborative efforts to protect critical assets and ensure system

security is also elevated.

Cognitive Task Analysis (CTA) is the process of extracting knowledge, thought process of cyber security experts and making use of this information to develop training scenarios (Huang, Shen, Doshi, Thomas & Duong, 2015). The outcome of CTA is the performance, equipment, conceptual and procedural knowledge used by experts as they perform a task. Training techniques for cyber security decision making will be developed. Informed by knowledge of mental models and their impact on SA, the research will

lead to new training techniques that result in transfer of skills and knowledge identified in this

research as critical to effective cyber security decision making. Measurement of mental models provides a way to evaluate structural knowledge and supports training and evaluation development; mental models that have been empirically developed from high performing experts can be used for evaluation in a variety of ways. Evaluating mental models can be used as a selection tool or a way to identify targets for training. To assess mental models that support cyber SA, it is important that measurement is well-suited to the mental model being assessed; because experts may hold multiple mental models, it is likely that several assessment techniques will be needed to assess all relevant mental models in CND. This training will be targeted to

two user populations: early career professionals, with the goal of improving human performance

in the industry, and students, with the goal of increasing the participation and preparation for cyber security careers.

Training materials is developed to teach novices how to perform like experts. In this paper, we present a cyber analyst training which is based on CTA approach to gain the insight of the cognitive workflow cyber analysts. Then, we find cyber analyst’s performance based on their response time of detecting cyber attacks comparing with estimated attack ideal time. Use of this assessment across diverse populations will demonstrate how cyber structural knowledge changes as a function of expertise. This research will identify patterns of gaps in structural knowledge within each population. It is expected that the most accurate and richest mental models will be held by cyber security professionals with the most industry experience. Even a different pattern is discovered, it will describe differences in expertise across populations. Ultimately, training needs for CND will be identified. This paper restrict scope of response time to the time taken by an analyst to conclude if a threat is a real threat or a false positive.

Literature Review

To understand and measure individual or team situational awareness and for evaluation of algorithms CyberCog (Rajivan, 2011), is used. CyberCog is a synthetic task environment for visualization intended to improve cyber situation awareness. CyberCog gives an interactive environment to directing human-tuned in examination in which the members of the investigation play out the tasks of cyber analysts. CyberCog produces execution measures and association logs for estimating individual and group execution performance. CyberCog has been utilized to assess group based situation awareness. CyberCog uses a collection of known cyber incidents and analysis data to build a synthetic task environment. Alerts and cues are produced based on copying of real-world analyst knowledge. From the mix of alerts and cues, trainees will react to identify threats (and vulnerabilities) individually or as a team. The identification of attacks are based on knowledge about the attack alert patterns.

Intended for better comprehension of the human in a cyber-analysis task, idsNETS (Giacobe, McNeese,  Mancuso & Minotra 2013), based upon the NeoCITIES Experimental Task Simulator (NETS), is a human-tuned simulator for interruption recognition analysis. Similar to CyberCog, NETS is also a synthetic task environment. The realistic scenarios are compressed and written into scaled world definitions and the simulation engine is capable of deciphering the scaled world definitions into a simulated environment, running the simulation, and responding to user interaction. In (Giacobe, McNeese,  Mancuso & Minotra 2013), several human subjects experiments have been performed using the NETS simulation engine, to explore human cognition in simulated cyber-security environments. The examination shows that the groups who had more comparative ranges of abilities showed a more firm cooperation by means of incessant correspondence and data sharing.

The primary difference between CyberCog/IdsNETS and LVC system ( Live Virtual Constructive (Varshney, Pickett, & Bagrodia, 2011) is that while CyberCog and IdsNETS are synthetic task environments, the LVC structure is a real system/emulator. A synthetic task environment may rely on previous incidents to generate the sequence of alerts and cues corresponding to those incidents, The LVC framework is able to simulate previous incidents as well as generate new simulated or emulated incidents on the fly (Huang, Shen, Doshi, Thomas & Duong, 2015). The LVC structure underpins a crossover system of real and virtual machines so assaults can be propelled from a actual or a virtual host, focusing on a real or a virtual host. Figure 2 outline the use instances of the LVC structure that consolidate physical machines and virtual system condition to perform cyber attacks and defense.

The Rationale and Objectives of the Study

The research objective of this proposal is to identify cognitive outcomes associated with

successful threat response in computer network defense (CND) and leverage those outcomes to

improve training for cyber security professionals. The role of cyber security professionals, who are responsible for ensuring the continued security of the network of their organization, has developed as a specialist subset in the careers of information technology. Broadly, cyber

security professionals investigate network activity to find, identify, and respond to anomalies.

CND is a joint human-machine collaborative task in which people depend on automated tools to

perform their jobs but must remain “in the loop” as an information processor and decision

maker. Consequently, CND is dependent on human decision making. Situation awareness (SA) and mental models are cognitive outcomes that predict human performance.

The research objectives of this proposal are to identify cognitive outcomes, including

mental models and situation awareness, that predict successful threat response in CND and to

create training to facilitate these outcomes. This proposal will address this objective through a

research approach that bridges human factors psychology and cyber security. Also, the objective is to improve the user experience of a training simulation model for a novice cyber security analyst to teach him how to think and act like an expert using characterization of cognitive analysis of a cyber security expert.

Research results will increase access to cyber security careers through the development of training for cyber security professionals and aspiring cyber security professionals, especially members of under-represented groups, as part of the educational objectives of these research. The recipients of this training include high school students. In addition, a new course will take an interdisciplinary approach to human decision – making in CND and expose students of computer science and psychology to the role of decision – making in CND.

Despite the presence of an interdisciplinary Human Factors M.S. program accredited by

the Human Factors and Ergonomics Society, students in traditional computer science paths receive limited exposure to human-centered approaches to technology problems, especially those incorporating science of decision making. Simultaneously, students in research psychology programs receive limited exposure to engineering applications of psychology. This new course will address this need. The course will be targeted to students majoring in computer science, psychology, and interdisciplinary human factors graduate programs.

As part of the educational goalsof this research, research outcomes will increase access

to cyber security careers through the development of training targeted to cyber security professionals and aspiring cyber security professionals, especially members of underrepresented

groups. Importantly, recipients of this training will include secondary school students. Further, a

new course will take an interdisciplinary approach to human decision making in CND and expose computer science and psychology students to the role of human decision making in CND.

The intellectual advantages of this proposal include new knowledge in the training science. The research will generate knowledge about the predictions of SA and performance in dynamic environments. The broader impacts of this project address the great need for the development of cyber security staff. Training in cyber security decision – making will make CND careers accessible to people who go beyond traditional careers in computer science. Threat

response training for CND will provide a strategic advantage, not only against known threats,

but against cyber adversaries as they continue to grow in sophistication and new threats emerge. Further, the training developed through this research is potentially transformativein that it will improve human decision making in CND, leading to better threat response and improved cyber security. Threat response training that improves the decision making skills in CND instead of training responses to individual threats will provide a strategic advantage against cyber adversaries as they continue to grow in sophistication and new threats emerge.

The Methods and Procedure

We propose realistic training scenarios for training and evaluation of cyber situations that allow cyber analysts to experience cyber attacks and learn how to detect ongoing cyber attacks. Cyber security lessons designed to involve cyber analysts in learning need to be carefully planned. We learn how, when, where and why to perform a cyber defense task. This knowledge can be used in the design of cyber security training scenarios to determine whether the attacks are real or false positive attacks. They would soon be overwhelmed by enormous data and would be forced to ignore potentially important evidence that introduces errors in the detection procedure. To solve the enormous cognitive demand faced by cyber analysts, we identify and design items on the cyber attack list. Cyber analysts can tailor their own watch list items and triggering thresholds in order to detect cyber attacks faster. Through collaboration with industry partner Cisco Systems, Inc., a provider of network solutions, cyber security professionals will be recruited as evaluators of candidate training products. In doing so, these cyber security professionals will benefit from state-of-the-art training in cyber security decision making. From this collaboration, a training workshop will be developed for early-career cyber security professionals. This workshop will introduce learners to the determinants of quality decision making in their careers, leverage the research to support development of cyber security decision making skills, and provide learners with methods of evaluating cyber security decision making.

Based on the design steps, the training workflow is shown in Figure 1, which contains the following steps:

Step 1 : The instructor shows the cyber security training scenario including an instruction sheet to describe the objective of the study. It includes expected time to identify the attack.

Step 2 : The simulated attacks and log data are shown to the analyst side. After analyzing these data, cyber analyst should react to these cyber events and identify in case of an attack or a false alarm.

Step 4 : During training, the training system can determine whether the cyber analyst’s response actions follow the expected time listed in the instruction sheet.

Step 5 : Based on the response time recorded for analyst in comparison with expected time, scoring is done. This analyst scoring is provided to the analyst for his next round.

Step 6 : Cyber analysts are asked to change their watch list items or based on their score report, they can improve upon the analysis capability.

Based on the customized learning scenario, cyber analyst will learn the necessary knowledge to monitor network conditions and to identify ongoing attacks. After cyber security training, cyber analysts can do the following with regard to a certain number of known attacks: List the relevant parameters for monitoring and knowing their characteristics in normal and abnormal operations. Recognize network attack symptoms. In particular, cyber analysts can isolate common network characteristics under attack and distinguish the specific characteristics of each attack (Huang, Shen, Doshi, Thomas & Duong, 2015). Given a certain number of current conditions (monitored parameters), you can analyze which type of attack occurs and how the attack started. Demonstrate proper remedial action procedures, including the selection of countermeasures to be applied and where to use them in the network.

References

• Tyworth, M., Giacobe, N. A., Mancuso, V., & Dancy, C. (2012). The distributed nature of cyber situation awareness. 2012 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support. doi:10.1109/cogsima.2012.6188375
• Giacobe, N. A., McNeese, M. D., Mancuso, V. F., & Minotra, D. (2013). Capturing human cognition in cyber-security simulations with NETS. 2013 IEEE International Conference on Intelligence and Security Informatics. doi:10.1109/isi.2013.6578844
• Mahoney, S., Roth, E., Steinke, K., Pfautz, J., Wu, C., & Farry, M. (2010). A cognitive task analysis for cyber situational awareness. PsycEXTRA Dataset. doi:10.1037/e578652012-003
• McNeese, M. (2000). Situation Awareness Analysis and Measurement. doi:10.1201/b12461
• Varshney, M., Pickett, K., & Bagrodia, R. (2011). A Live-Virtual-Constructive (LVC) framework for cyber operations test, evaluation and training. 2011 – MILCOM 2011 Military Communications Conference. doi:10.1109/milcom.2011.6127499
• Huang, Z., Shen, C., Doshi, S., Thomas, N., & Duong, H. (2015). Cognitive Task Analysis Based Training for Cyber Situation Awareness. Information Security Education Across the Curriculum IFIP Advances in Information and Communication Technology,27-40. doi:10.1007/978-3-319-18500-2_3
• D’Amico, A., Whitley, K., Tesone, D., O’Brien, B., & Roth, E. (2005). Achieving cyber defense situational awareness: A cognitive task analysis of information assurance analysts. PsycEXTRA Dataset. doi:10.1037/e577392012-004
• Rajivan, P.(2011). CyberCog:A Synthetic Task Environment for Measuring Cyber Situation. Master Thesis of Arizona State University

Tables and Figures

Figure 1. Workflow for training system

Figure 2. This usage example of Live-Virtual-Constructive (LVC) framework adapted from Military Communications Conferencepaper.