The technology bets on process improvements aligning product and tech organisations with business realities for superior outcomes compared to incremental improvements in TCO, development velocity, or application performance.
We’ve all heard the adage ‘change is the only constant’, which holds particularly true in the fast-paced world of IT. But often, while innovation may be stated as the intent of a digital transformation undertaking, the underlying justifications are flimsy at best. Sure, the team may offer some rationalisation around improving build times by 20%, reducing cloud costs by 10%, or halving the lines of code for new features, but I can’t help but wonder if, many a time, it’s just a fear of missing out.
Risk outweighing reward?
Sometimes these investments pay off big, overcoming obstacles holding the company back. The right tech initiatives unlock scale and resiliency, which may be the difference between the business becoming viable. Far more frequently, the juice isn’t worth the squeeze. One commonly cited estimate suggests that 70% of tech projects fail. While these aren’t all related to software development, something like a two-thirds failure rate feels right. These experiences make us incredulous about the marketing claims around new technologies. How often have we taken significant risks to incorporate new tech only to find the integration was far more challenging than expected and the improvements far less than promised? Often, we regret these changes. We would have been better off spending the lost quarter bolting on new features to the old systems. At least then, the sales team would have been much happier.
Do you dare to stream
Given the risks and expense, how do tech leaders and architects reason about the return on investment (RoI) of event streaming? Product and technology organisations are notoriously bad at rationalising return on investment. Future development acceleration is among the most common potential benefits of new technology, but it’s hard to quantify and predict. Even benefits the CFO assures us can be calculated in the abstract, such as TCO, which become difficult to measure in real organisations. Sometimes, RoI is justified in terms of promoting agency within the teams. At least once, I’ll admit I’ve authorised a new tech stack as an excuse for removal. I wouldn’t accept another busted sprint that the team blamed squarely on the legacy application.
Event streaming may have a lot of hype, but one can’t compellingly argue that it hasn’t changed the world. Activities most of us do daily, like ordering packages from Amazon, using a credit card, or calling an Uber, have us interacting with systems that depend on event messaging systems. That doesn’t necessarily mean that event streaming makes sense for every use case.
Real-time, Real Value
So, how does one decide if it makes sense in your enterprise? Practitioners often cite “real-time” as streaming’s chief virtue. Yet, event streaming may still be worth the investment, even if reducing the latency between data generation and insight isn’t that important. I’ve met many who dismiss streaming as just a faster “batch” that should only be used when low latency is required. I’ve come to believe this is the same fallacy Microsoft made when it simply saw mobile as a desktop experience with a small screen. Apple instead saw mobile as a new paradigm that would transform how people interacted with technology, and it won.
Similarly, many incumbents didn’t recognise the internet’s potential. They were displaced: from newspapers seeing the Internet as just high-margin distribution to shops only seeing it as a new channel to sell existing inventory. Paradigm changes don’t offer a faster, better world of yesterday but a fundamentally new way of doing things. History is full of examples of companies that embrace the new paradigms to beat those that just see faster horses and smaller devices.
Event-driven architectures are a similar enabling capability. They create new opportunities and approaches to building software and structuring technology teams. In particular, streaming’s reduced time-to-market for new insights is transformational. As companies become software, we can’t anticipate every use of data in our products a priori. With event streaming, a new product or insight is often a matter of building a new consumer group rather than needing to break apart an existing data model. The bigger data gets, the more specialised one must be in how it is organised and queried. In event streaming, the specialisation is that data is optimised for consumption rather than a particular query pattern. More than other paradigms, it supports new ideas and product features. This creates another important effect: by decoupling applications from bespoke point-to-point communication paths, dependencies between teams become better specified and scalable. Event messaging provides a foundation for different teams to produce events than those that consume them.
A safe bet
In my experience, the technology bets that foster process improvements that better align the product and technology organisations to respond to business realities yield far better outcomes than those that are better mousetraps that incrementally improve TCO, development velocity, or application performance. And that may be the most important RoI consideration for deciding if event streaming makes sense in your organisation.
