On September 14, we (specifically Mason) participated in an interactive discussion hosted by the Global Cyber Alliance (GCA) [https://www.globalcyberalliance.org/] with the topic, SMB Focus Group 1: Implementation and Barriers to Cybersecurity. Only four IT service companies were in attendance, including us.
This is an issue that is near and dear to our hearts because we are small business, and we serve small businesses. Our entire service model originated with the observation that small businesses are at a disadvantage in their cybersecurity needs. Because small businesses do not have the same buying power as large enterprises, they do not have access the same tools as are available to enterprise. Even if SMBs could access the same tools, who in small business has the expertise to use those tools effectively? Almost none of the software programs and services that small businesses use are designed with security at their core. Further, if there are any security controls available they are probably disabled, and the small business owner is expected to know which ones to turn on and why. Obviously most business owners are not experts in cybersecurity so they become sitting ducks for cyber criminals.
What Is The GCA And Why Does This Matter?
According to their website, “The Global Cyber Alliance (GCA) creates and equips communities to deliver a more trustworthy Internet for all.” GCA’s partners include the RCMP, law enforcement agencies from several other countries, security software providers, financial institutions, and on it goes. We feel that having the ear, even for a short time, of an organization as influential as GCA is one of the best ways to spread awareness of the issues we see “on the ground”
Mason ended up exchanging several emails with the event host, and one email that Mason sent seems to have really resonated. We have no idea what goes on behind the scenes, but several recent communications from both the GCA and the US Cybersecurity & Infrastructure Securities Agency (CISA) appear to echo same ideas that Mason presented in the focus group and later in email. It seems like we’re onto something.
This month (October 2023), GCA published a report that echoes many of the points that Mason made during the focus group and in subsequent communications with the focus group organizer. You can download the full report here: https://www.globalcyberalliance.org/reports_publications/defensive-measures-against-ransomware/
The report is sobering if you are in small business. The authors (correctly) identify that small businesses are typically under-defended, stating, “most small businesses fall below the ‘Cybersecurity Poverty Line’, identified as ‘the line below which an organization cannot be effectively protected.’” It continues by asserting that “[t]he typical small business network is built at low cost to support necessary business functions, without considering security as a design criterion.” Both of those observations match our observations too. The result is that we have a huge segment of the national economy propped up by businesses that do not know how to put basic controls in place to defend themselves.
In the email below, Mason articulates what we see as the problems businesses face and what small businesses actually need. The full text of that email follows below, with minor edits to add visual interest, remove any personally identifiable information, and remove most references to specific tech corporations. The email is long, and the intended audience is a technical reader. So be warned, it could be a tough slog. However, it is worth reading if you want to understand how you have probably been mislead about whether your critical business data is actually safe right now.
Hi [event host],
Thank you very much for facilitating the discussion the other day. I enjoyed the opportunity to share some of the challenges facing this massive, but dramatically underserved and misunderstood market. There wasn’t remotely enough time during the call to really get to the heart of the challenges that small and micro businesses face, so I want to share some further thoughts with you, your organization and hopefully your partners. My hope is that by sharing my perspective, that perhaps we can start talking about how to make a material change in the cyber security of the very small organizations that make up an inordinately large percentage of the employment in our countries, and by doing so also improve the health and resilience of the technology companies that are best poised to make this change.
I would like to preface this discussion by saying that discussions about cyber security topics, both within and without the cyber security community, tend towards discussions of absolutes, sweeping statements, minutia, corner cases and typically contain strong opinions. In short it tends to be fairly extreme. I fall prey to this kind of talk as well, which I think is rooted both in the personalities of people in this industry, but also in the frustration of seeing an unending stream of preventable disasters and being ignored by decision makers. Thus, you will see some strong opinions in this email, but hopefully you’ll also see an attempt at addressing the problems in more constructive ways that more directly address observed behaviours.
How do we define small businesses?
According to Wikipedia, small businesses are defined very differently across developed nations. “Small businesses range from fifteen employees under the Australian Fair Work Act 2009, fifty employees according to the definition used by the European Union, and fewer than five hundred employees to qualify for many U.S. Small Business Administration programs. While small businesses can also be classified according to other methods, such as annual revenues, shipments, sales, assets, annual gross, net revenue, net profits, the number of employees is one of the most widely used measures.”
In North America, the separation of businesses into small, medium and large, is not granular enough. There are very good reasons to believe that these delineations are too coarse and that these arbitrary groupings result in the overwhelming majority of businesses being largely invisible to government and non-profit cyber security programs. I would argue that we need to at least include a 4th micro category, which some government articles do refer to, albeit not as a standard categorization.
What is the scale of the contribution that small businesses make to our economy?
Canadian federal government stats
- As of December 2021, there were 1.21 million employer businesses in Canada. Of these, 1.19 million (97.9%) were small businesses
- In 2019, small businesses contributed 36.7% to gross domestic product (GDP) generated by the private sector
- My company’s target market of 5-50 employee businesses represents 40% of the total number of businesses in Canada
- We are trying to work out how to serve companies of between 1-4 employees who make up 55% of Canadian businesses
- 68% of Canada’s private sector employees work for small businesses
- 15.5% of small Canadian businesses are in rural areas
Can we treat all small businesses the same?
Clearly small businesses are tremendously important to the Canadian economy and to the livelihoods of more than 2/3rds of the working population. What is often missed in stats like those above is that the majority of employment comes from the “micro” sized businesses. What I am going to attempt to convey in this email is that the tools, techniques and advice that make sense for medium and large businesses are largely inappropriate for micro businesses and even for most small businesses. I’ll also show that the supposed cost and security advantages of cloud services, which are enjoyed by larger businesses, do not apply to micro businesses and are even worse for rural small businesses.
My intention with this email is to attempt to encourage your organization and governmental departments such as NIST, CISA, and the Canadian Centre for Cyber Security, to see that a different approach is needed if you want to see a material change in the cyber security of 98% of the private sector businesses that make up the US and Canadian economies.
Opinion, from someone that has been on the frontlines with micro-organizations for over 20 years, as to the prospects for current tactics and strategy to make meaningful change
Micro businesses are largely not driven by data, analysis or facts. They are driven by the emotions and opinions of their owners – no matter how detached they may be from objective reality. The vast majority of cyber security guidance, from all levels of government and other well-meaning organizations, which is aimed at small/micro businesses, largely consists of mild FUD, highly generalized descriptions of issues, brief stories that are intended to make an emotional connection, vague technology checklists, lists of vendors, references to inscrutable standards and unrealistic calls to action. The root of the problem isn’t a lack of data, attractive presentation of that data, simplification of that data or scary stories to spur people to action, the root of the problem is that there are almost zero vendors who are trying to address the actual challenges that micro businesses face and the micro businesses are entirely incapable of implementing the guidance on their own. Micro businesses require a lot of hand holding, they have very little money, they can be irrational and erratic in their decision making (because they’re pulled in so many directions) and they have a justifiable mistrust of vendors which makes it very difficult to sell them what they actually need. From a sales perspective, if all you have to sell are point solutions and if you’re competing for a small slice of their IT budget, they’re generally not worth spending time on. There are some exceptions – printer leasing/rental companies seem to survive and do a passable job and the same is true for telephony vendors, but general IT vendors are a whole different story.
On the call yesterday I said that I have personally spoken with many hundreds of small and micro businesses. I managed a tech support team for a small rural cable ISP which, in addition to the residential customers, served 400 micro and small businesses, including the municipal governments of 3 communities. I was a sales engineer for a much larger cable ISP (Shaw) where I supported eight small business outside salespeople who covered most of BC (except for Vancouver and Vancouver Island). I was also a sales engineer for the education, municipal and enterprise channel in BC, where I spoke with dozens of small municipalities, regional districts, school districts, universities, and medium enterprises. I was also a sales engineer that supported Shaw’s partner channel, where I worked with more than a dozen IT and voice service providers in BC. It was my experience in these roles that opened my eyes to the experience that most small organizations face (private, public and non-profit) when it comes to their IT infrastructure. I generally sum up the quality and nature of IT service provider support of micro and even small businesses as being exploitive, incomplete, incompetent or all of the above.
From listening to stories from my tech support staff, salespeople and my own conversations with hundreds of small/micro-organizations, I have learned that they all suffer from the same problems and all share common stories around business continuity and security challenges. It doesn’t matter whether the business is a manufacturing company, professional office, non-profit or municipality, the stories are the same. They all feel overwhelmed and intimidated by technology; they all have stories of paying unnecessarily large sums of money to a vendor or service provider; so many of them have experienced major/catastrophic data loss due to their own understandable ignorance or the incompetence of their service provider; so many of them have fallen prey to BEC scams, ransomware, worms (back in the day), banking trojans, etc.; and so many of them have had major and costly downtime related to inadequate infrastructure and lack of a viable business continuity plan. In the majority of cases, the business owner or manager will say that they didn’t see this coming, they see it as a freak accident, they don’t know anyone else who this has happened to, and they didn’t think it could happen to them. From my perspective, I see a recurring pattern of behaviour between business owners/managers and service providers and most importantly I see that this is wholly unnecessary.
We’re always told to learn from history so that we don’t repeat the mistakes of the past. Unfortunately, we generally don’t seem to heed that advice. I want to look specifically at the recent (last couple of decades) history of on-premises infrastructure and the current trend towards cloud hosted services (specifically SaaS), as this relates most closely to the challenges that small/micro businesses face. To start, let’s take stock of where we are today and look at the reality of SaaS apps, from the perspective of micro businesses. We’re finding that as software vendors are increasingly dropping their on-premises offerings and forcing customers to use SaaS, that we are encountering six major challenges.
The first issue is that, almost without exception, vendors use security features to differentiate their packages. The “Startup”, “Team”, “Business Basic”, “Essentials” or other marketing term for the SaaS provider’s entry level package often doesn’t even include MFA, it provides no means to enforce a defensible security policy and it absolutely will not include any integration with an external IdP. Thus, any thought of SSO or any other way of reducing attack surface or password sprawl is moot. In one fell swoop, all defense-in-depth has been stripped away and we’re left hoping that the user does the only thing that can protect their data – good password security hygiene. Twenty years of industry experience shows this is highly unlikely to happen if left up to users.
The second issue is that most smaller SaaS apps (and this includes huge services like a well-known documentation platform, as well as major accounting services), do not give you the option of having an unlicensed admin account. Every account is a paid account. There are two problems with this. First, for a one-person business, buying a second license doubles your subscription costs. Even for a 10-person business, buying a separate admin license still increases your subscription costs by 10%. Whereas in very large companies, purchasing a separate admin license costs almost nothing. The second problem is that within the UI and support documentation, the expectation is that an individual user will be made a member of the admin group, so that they can administer the application. They of course will also use this account for their day-to-day work in the application. They’ll also use this account, from their work PC, their personal smartphone and their home PC, because SaaS marketing has trained them to demand mobility. This is also the account that they’ll share with their MSP, which will lead to all sorts of problems with MFA when the MSP attempts to help the customer administer their application. On the LAN, we long ago learned not to give regular users admin rights in an AD domain, but the move to the cloud seems to have erased that knowledge and now it’s happening all over again, but this time with a dramatically increased attack surface.
The third issue ties into the first and second. The adoption of SaaS typically results in a reduction in the security of our customers, through a loss of defense-in-depth, a loss of visibility and a massive increase in attack surface. If you want to restrict access to your SaaS tenant by IP, client TLS certificate, VPN, private interconnection, static IP, reverse DNS name, etc., you must pay for an “Enterprise” service tier. The fact that Microsoft was recently publicly shamed into making Entra ID logging data available in all service tiers, as opposed to only in the enterprise packages, highlights this trend (https://www.microsoft.com/en-us/security/blog/2023/07/19/expanding-cloud-logging-to-give-customers-deeper-security-visibility/). I’ve never met a small business that would willingly pay $30+ per seat for an enterprise license, let alone pay for multiple enterprise licenses for multiple SaaS apps. The result is that the only security “knob” that we have to twiddle is password strength, thus all the defense-in-depth that we can cost effectively deploy in an on-premises environment evaporates when customers go to SaaS. We typically cannot enforce MFA for SaaS because the services have no mechanism for that or they hide it behind a paywall, so our customers are essentially opening up their applications and data to the entire Internet. So, when CISA made a statement to the US government that all businesses should adopt a zero-trust model and that legacy applications should be like SaaS apps that are designed to work securely across an untrusted network, I almost choked. This is not even an aspirational goal; this is pure fantasy. This leads me to the fourth issue we face.
The fourth issue concerns the technical and organizational ability of SaaS vendors to secure their applications. The largest vendors are not doing a great job of securing their SaaS offerings, especially where small businesses are concerned. Small ISVs, like the dental software companies that I mentioned on the call, are woefully ill equipped to be running their own SaaS offerings. Their developers are not trained in security, they don’t have dedicated security staff, they have never employed operations staff who know how to manage a SaaS service, so many of them are just taking their 20yr old Windows application, that they would normally spin up on a LAN, and they’re hosting it in a Windows VM on AWS or Azure and then sticking a web front end on it. When I ask them to provide some assurances about the security of their infrastructure, so that we can tell our customers how their data is being protected, they respond that they will not share proprietary information. So, we’re completely in the dark, other than we know how bad their security was with their on-premises solution, so we assume their SaaS offering isn’t much better.
The fifth issue concerns business continuity. There are three parts to this issue: Reliable Internet, cost and backup/recovery
The first is that once a business moves their data to the cloud, their Internet connection must be reliable enough to maintain access to that data. I worked for cable companies for 15 years and built, managed and maintained a lot of cable infrastructure – it’s not very reliable. I worked as a sales engineer for a larger carrier who purchased a lot of tariffed data services (DSL, T1, fixed wireless, metro ethernet) and I have seen firsthand how poorly so many of these wholesale services behaved once installed. I have also spent a lot of time selling data services in rural locations (i.e., all of BC except for Vancouver) and I can tell you that many ISPs have no redundancy available in most rural communities – that is there are many stub networks that can extend for hundred of kilometers and which service hundreds of thousands of homes and businesses – I have seen a single car crash burn through the aerial fibre that services dozens of municipalities. Even across carriers, in rural locations they will all piggyback on the same fibre cable, so there’s no increase in redundancy when you add an LTE backup to your cable/DSL Internet service.
The second business continuity issue is one of cost. Even if a micro business has access to truly redundant Internet connections, adding a second Internet connection is often doubling their monthly IT bill. Therefore, on top of all the issues listed above, the micro business is more exposed to business continuity problems when their data is no longer local.
The last business continuity issue relates to backups. Again, the way that SaaS services have been marketed has led small/micro business owners to falsely believe that once data is in the cloud it’s safe. The reality is that very few SaaS providers actually back up a customer’s data. They may spread it around through availability zones, so that it is available and not subject to hardware failure, but that’s like saying that RAID is a backup technology – it’s not. Most SaaS services, including massive ones, have limited mechanisms for performing automated backups of data (I know because we offer a backup service for one of the largest SaaS applications, and the vendor’s APIs do not provide the ability to backup up certain data in the chat application or even certain parts of the mailboxes) and some others offer no means of doing automated backups at all (just manual backups). Restores can be even worse than backups because SaaS vendors don’t seem to feel the need to maintain backward compatibility as they evolve their services. Also, if a SaaS vendor goes out of business, even if you made backups, there’s no way for a micro business to migrate that data to a competitive service, because the backup is almost certainly not in an open format or one that can easily translate to another service. Essentially, everything that we learned about backups and open standard file formats has gone right out the window.
The sixth issue relates to quality of support and financial sustainability of the service providers that directly support these small/micro businesses. SaaS providers, especially the largest ones, want to disintermediate their former local IT partners. They want to sell directly to the end users and have those end users support themselves through context sensitive help, community forums, and knowledge base articles. When users call for support, it’s all billable and the cost and quality of the support is usually terrible. The largest software vendors will allow small resellers to make single digit margins, yet due to their own abysmal support services, they effectively push the support burden onto their resellers. In short, they now want to be even more monopolistic.
It’s important to reiterate that this whole section is related to SaaS, not IaaS or PaaS. However, even though IaaS may be more mature and can address some of the issues I have posed with SaaS, it is absolutely cost prohibitive for micro businesses.
All-in-all, I’m wildly pessimistic about the ability to meaningfully defend SaaS apps for small businesses. It just seems that we’re making all of the same mistakes all over again. The cyber security community eventually did a fairly good job of identifying what kinds of attacks are successful against a typical LAN environment (Mitre ATT&CK), we developed best practices and easy to implement checklists (e.g., CIS Critical Security Controls) and the vast majority of businesses and IT providers completely ignored them. The deplorable state of on-premises security was exacerbated by some truly egregious and monopolistic behaviour on the part of certain vendors, a lack of quality educational resources or standards for technicians and sysadmins and the general greed and excessive risk taking that goes on in our capitalist societies. As a result of failing to follow our own hard-won guidance or to deal with a monoculture of incredibly weak software, we decided that on-premises infrastructure was “bad”, impossible to defend and thus it needed to be replaced. Compared to our failures on the LAN, cloud services seemed like such a great idea; we were told that “competent people will be contractually and commercially liable for protecting your data! It will exist outside your flawed on-premises network and so you won’t have to worry about it.” Instead, the move to SaaS apps looks more like we’re running backwards at top speed, not confidently walking forward. We still largely have the monoculture of weak software at the edge of the network (PCs), but now we’ve connected these devices directly to the Internet and have placed unwarranted trust in these devices to serve as trustworthy proxies for user identity. We’re still using passwords 20 years after the security community declared them to be dead. Not only are we using passwords, but we’re still storing them on servers rather than at least using technologies like SRP to remove some of the attraction of stealing databases filled with hashes of weak passwords. The fact that we rushed headlong to the cloud and that the world’s most profitable companies didn’t attempt to build their services on something stronger than shared passwords, might be considered by some to be criminally negligent. On the server side, we now have an impenetrable fog obscuring visibility into what’s actually going on with our data; we know that the global security industry is short about 1.4 million security professionals, so where are these competent people that are protecting our data in these juicy, large, centralized targets; SaaS providers still have PCs and employees that are just as phishable as the rest of us; the attack surface is now global; we have a 10 trillion dollar cybercrime industry that is happy to bang away on this globally reachable attack surface; we have security “researchers” that publish tools later used in attacks by criminals. And how are we responding to the ever-escalating criminal activity? All levels of government and the commercial sector are flocking to the cloud and allowing anyone anywhere to access web applications; we’re trying to identify bad behaviour with AI/ML; we’re creating bug bounty programs to encourage people to attempt to break systems and report the flaws; we’ve made hacking cool by supporting BlackHat, creating the pen testing industry, running capture the flag competitions and making a lot of movies about cyber attacks; and we’re again pushing user security training with the ridiculous premise that users are going to be able to “spot the phish” when the phish was generated by ChatGPT and their mobile device doesn’t show a URL, email headers or any validation related to the useless little padlock. I recommend that everyone in the security industry read Marcus Ranum’s “The Six Dumbest Ideas in Computer Security” that he wrote back in 2005. Marcus’s words are far more relevant today than they were in 2005. We’re not learning. We’re not getting better. We just keep investing in more layers of dumb ideas rather than getting the basics right and addressing root causes.
What do small/micro businesses actually need?
I have learned that micro businesses and to a large extent small businesses have very similar needs. When compared against organizational maturity models and IT maturity models these organizations are like children. As we do with children, we need to protect them, nurture them and train them while they are in their immature state. Even once they attain a level of maturity, they still need protection from forces much larger than them.
- Micro businesses and most small businesses do not have, and cannot hire, the in-house knowledge and experience to competently assess security and business continuity risk. As a result, if they are to adequately defend themselves, they need to be insulated from having to make these sorts of decisions. This requires that they have:
- A relationship with a trustworthy partner, who has the business’s best interests at heart and who will make appropriate security and business continuity decisions for them.
- A highly opinionated service that addresses foundational security and business continuity challenges in a comprehensive, integrated and cost-effective manner, appropriate to the size of their business. Security and business continuity are not add-on or optional features; they are foundational and designed in.
- Ongoing training in positive feedback style security behaviours, which are consistent over time and build toward increasingly mature security processes. This is in stark contrast to the negative feedback style of training people to always be vigilant and watching for suspicious behaviour.
- Micro businesses are easily swamped by fixed costs and cash flow challenges can make even minor capital investments infeasible. This means they do not invest in foundational systems early enough. By definition they do not have access to the revenue volumes and economies of scale that larger businesses rely on. Therefore, they need:
- A complete solution which covers all IT costs, and which is tailored to the financial realities of businesses of their size
- A solution that is delivered as a monthly recurring subscription, so that micro businesses are insulated from capital expenditures and fluctuating costs that can affect their tenuous cash flow
- A recovery oriented and failure tolerant solution, that ensures the availability and continued ownership of the organization’s data, even in the face of unreliable or inadequate Internet access and local hardware failure. This is significantly more important for rural businesses, where there is no such thing as overnight shipping. Internet access failures can sometimes take days to weeks to repair, and many locations have highly asymmetrical Internet service with less than 1Mbps upload speeds.
- The vast majority of SMBs do not need or want the constant churn of consumerized IT. This represents a significant drag on their productivity and security, and it increases costs.
- Businesses function best when they can optimize and refine systems so they build highly efficient muscle memory. I am constantly hearing from our customers that the constant change in applications is hurting their businesses, not helping it.
- Constant deployment of new features, typically configured in a wide-open way to increase adoption, represent an ever-growing attack surface for features that most businesses will never be aware of let alone use. IT providers either end up playing whack-a-mole to try to lock down these features or they too are oblivious to them and thus allow them to proliferate.
- Most businesses are best served by a simple set of core services that are refined over time, rather than being constantly replaced.
How can we provide what small/micro businesses actually need?
There are a vanishingly small number of vendors that are capable of delivering the kind of service outlined above. Yet we know this is feasible, because we’re doing much of this today. What has made this possible is that we understood that most small businesses were never built on a solid foundation, so we set out to install a foundation under their existing business. If you’ve ever seen an old house lifted up, a foundation poured and the house set down on the new foundation, that gives you some idea of what we do. Just as you wouldn’t want to invest money into a home that was not built on a solid foundation, because if the house shifts or sinks all your expensive additions will break, it doesn’t make sense to attempt to bolt-on security and business continuity to a business that does not have a strong IT foundation. While this sort of approach would likely be incredibly difficult for larger organizations, it is feasible for most micro/small businesses.
The need and therefore the opportunity to serve this vastly underserved market is enormous. These needs are not going to be serviced by gigantic corporations, because their structure precludes them from being able to establish the trusting relationships that small organizations need or to muster the local resources necessary to implement a comprehensive solution. What is needed is a focused, unified and prolonged effort to build local capacity in the communities in which small organizations operate and to prevent brain-drain into the few massive corporations that currently dominate the technology industry. This will require the sharing of business models, shared development of tools and services that can be deployed at the real edge of the network, governance of shared configuration standards and governance of standards for IT staff and service providers.
The Internet was designed to be decentralized in order to be resilient to attacks on major infrastructure. The Internet was designed to have smart edges and a dumb core, so that diversity at the edges of the network would lead to greater resiliency and innovation. It is far more secure and resilient to have simple, hardened, diverse edge locations, serviced by a diversity of small suppliers, than it is to build “too big to fail”, highly complex, opaque, centralized, self-interested systems, which amplify the risks of catastrophic and systemic failure. However, I’m not advocating for the other extreme of the anarchy and wastefulness of so-called fully distributed, trustless systems that are exemplified by cryptocurrencies. There is a need for hierarchical and responsible governance in order to provide the resources and structure to enable large scale planning, coordination and sharing of resources. It’s just that the hierarchy must genuinely serve the needs of those at the edge rather than competing with them, controlling them and exploiting them for their own ends.