organization hasn’t determined how effective a control is at mitigating a risk — the act of trying to automate evidence collection and the resulting evidence often informs future conversations about the effectiveness of a given control. This also isn’t fancy; this automation usually involves copying files or using an API call to automate data collection. Ideally, all of the evidence is stored in a centralized location, and the automation can notify the compliance team of a new piece of evidence, whether by email, or ticketing system, or something else. This is also an opportunity for the compliance team to build a bridge and partnership with the security team. While the security team will probably need to help set up the script, API call, or similar automation, if done well, this is a set-it-and-forget-it arrangement. Once evidence of a control’s operation is automated, the security team won’t be bothered for evidence again by the compliance team. This reduces organizational friction and allows the security team to focus on their remit, which is securing the company. Bolstered by this, the security team may be very amenable to helping to set up more automations, as it means less busywork for them in the future.
Step 3: Automate Control Testing
Although it’s tempting at this point to assess how effective each control is at mitigating the linked risks, the next step is instead to automate testing the control effectiveness as much as possible. These are simple rules. For example, if the vulnerability scanner failed to run last week, the automation should notify a human on the compliance team, assuming that there’s a policy requirement or similar
that mandates the vulnerability scanner be run weekly. Similarly, if a piece of evidence was expected from a control but doesn’t arrive at all, the compliance team should be notified. The goal is that when a test fails that an issue is identified and logged by the compliance team and tracked to resolution, preferably in an existing ticketing system that the IT or security team uses already.
Not all controls support automated testing, however. For example, consider a control that requires senior management to demonstrate a commitment to security, and that commitment is measured by their having signed one or more policy documents within a defined timeframe. The evidence collection — copying the latest signed files from one location to another — should be automated or scripted, and a notification should be sent to a compliance analyst to review the files. All that’s required is for the analyst to evaluate who signed each document, and if they have adequate organizational authority to demonstrate that commitment.
Step 4: Estimating Control Effectiveness at Mitigating Risks
Now that the organization has linked controls to risks and is automatically collecting and evaluating as much evidence of control effectiveness as feasible, it’s possible to apply some simple rules to estimate how effective each control is at mitigating linked risks. First, the organization needs to determine if a control is healthy. You can determine if a control is healthy if it has regularly updated evidence that shows the control is effective.