[ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate
[ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate [ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate
Why Service Consumers and Service Providers Should Never Directly CommunicateDocument ID: ZAPFLASH-2008228 | Document Type: ZapFlash
By: Ronald Schmelzer
Posted: Feb. 28, 2008
Edward R. Murrow once said, People say conversation is a lost art; how often I have wished it were. How true it is in the vendor and media-spun world of IT and even more so in the case of SOA. Many people think they are doing or talking about Service-Oriented Architecture (SOA), but most of the time theyre really doing Web Services Integration (WSI), developing Just a Bunch of Web Services (JBOWS), or even just point-to-point XML (what many would like to call REST, but even in most cases is not even truly Representational State Transfer [REST], which is also an architectural form). Even after 8 years of evangelism, research, advisory, and deep consulting, were still finding a large percentage of so-called enterprise architects pursuing SOA projects as if they were point-to-point integration projects.
Weve discussed many times from an economic, technical, architectural, and philosophical perspective the problem with not understanding that SOA is an architectural concept, so I wont bore or nag you further in understanding that SOA is not just another name for integration. However, we need to crack this nut and crack it definitively if this industry is to make progress from its current inefficient, ineffective, unpredictable, overly-expensive, overly-complex, and rudimentary form to the predictable, reliable, cost-effective, disciplined, and easily measured part of the business organization that we know it should be.
While weve made our pleas to the business, architectural, and process-driven parts of the organization and to vendors in previous ZapFlashes, we want to focus this ZapFlash on where the problem specifically lies: within the IT organization, and specifically the part that concerns themselves with so-called integration issues. Folks (and Im speaking directly to you integration-centric techies), you are getting in the way of making SOA a success probably due to your own misunderstandings and misconceptions of how SOA should work. And to correct your misunderstanding, I need to speak your language: the language of integration technology.
The Problem of Service Consumer to Service Provider Communication
In typical integration environments, youre primarily concerned with the movement of data or the amalgamation of business logic to achieve a specific task. In that vein, most integration developers think about inputs (gazintas), outputs (gazattas), transfer, and transformation. That is, they think about how theyre going to get data or application functionality out of one system, how theyll get it into another system, the mechanism theyll use to get that information or functionality from one place to another, and then what they need to do with it while its en-route to transform it from one format, protocol, context, process, or policy into another. So, while architects are chemists that is, scientists who are trying to understand a system and how it can all be put together, integration developers are chemical engineers focusing on just the mechanisms of transformation.
developers see the concept of the Service as just another means of gazinta or gazatta. That means these integration-heads need to find something else to do transfer and transformation. Vendors to the rescue! Most vendors responded to this desire by simply dusting off their existing integration middleware. End-users responded easily to this insignificant change and voila, now they have transfer and transform that can deal with their Service-based gazintas/gazattas. But does this make SOA? Oh no, no, no. But weve already beaten you on the head with that a million times. So, lets take a different tack. The reason why this approach is wrong is because Services Consumers (what you might think of as a gazinta) should never communicate directly to Service Providers (what you might erroneously think is a gazatta).
>From this integration-centric mentality, most integration
Why? First, the main concept of SOA is that we want to deal with frequent and unpredictable change by constructing an architectural model, discipline, and abstraction that loosely-couples the providers of capability from the consumers of capability in an environment of continuous heterogeneity. This means that we have to somehow develop capabilities without the knowledge of how they might be used. As we delved into the Seven Aspects of Loose Coupling in a previous ZapFlash, this means that we need to have significant variability at a variety of aspects including the implementation, infrastructure, contract, process, policy, data schema, and semantic aspects. Having this variability allows companies to have stability in their business and IT even though the IT and business continue to change. Agility, predictability, reliability, visibility, and cost-effectiveness all become that much more realistic when we can achieve that sort of abstraction.
But to achieve that abstraction means that we cant afford things to break when things change. Lots of things change in an IT or business environment: location and availability of services, versioning of Service implementations and contracts, changes to business processes and policies, new or changing requirements for reporting, visibility, and control, and of course the need for new capabilities to solve emerging business problems or capitalize on new business opportunities. However, if you think that a Service is just an API, then all hell breaks loose when you try to connect a Service Consumer directly to a Service Provider. What if the provider moves? What if the Service contract changes? What if the provider is no longer available? What if you now need a new communication mechanism or change to a data schema?
The first knee-jerk reaction to this complexity by most integration architects is to put some piece of technology between the Service consumers and providers to ameliorate this problem. The impulse is correct: you cant have direct communication if you want to have variability. However, the execution is all wrong. If you are relying on black-box, and often proprietary, technology to do the heavy lifting of managing Service communication differences youll simply shift all the complexity and work from the end-points (the Services) to an increasingly more complex, expensive, and brittle middle point. Welcome to EAI 2.0, the secret and unpublicized real name for most so-called Enterprise Service Bus (ESB) products or at least the way that many are using them.
(Re)Introducing the Service Proxy
So, if reworked middleware isnt the solution, what is? Well, of course the answer is architecture. But thats too pat of an answer, so to speak again in the parlance of integration developers, lets explore how a system would be designed if you cant count on any particular vendors solution to be in the middle to make it all work.
Lets say you want to build a Service Consumer to consume or compose some functionality, but you dont know where that functionality is or even how to communicate with it. There are at least three ways of handling this particular challenge. First, you can talk to a proxy (or broker or intermediary) in a language that you do understand and thats in a location that you already know and send a request to this proxy that will act on the behalf of the Consumer. This is the envelope within an envelope concept. Basically, you enclose a message to deliver to someone whom you know, but dont know where they are, within an envelope that is addressed to a party that you do know. That party then opens the envelope and addresses the internal envelope to the end recipient based on a look-up of some sort. This approach simplifies the problem of the Service consumer having to know about changes since they just need to know about the proxy and the end-recipient. From a technology perspective, the use of WS-Addressing simplifies this approach. In essence, a Service consumer just needs to know the WS-Address of a Provider and hand that to a Service proxy to resolve and deliver.
Weve discussed this concept of the Service Intermediary before. The problem with this approach is that the Service proxy still needs to find a way to deliver the message, and if the proxy contains all the rules internally in the black-box, we have the same problem we have with EAI 2.0-style ESB solutions. This is where the registry steps in. Service proxies should not store their own rules, but rather get all their routing, location, and policy-based binding rules from a central system of record. In SOA, this is the role of the registry. Proxies can cache the results of registry look-ups for performance enhancement, but the metadata themselves should come from the registry.
So, registry-configured Service proxies solve many of the problems of Service communication, but it still leaves open the problem of the connection between the Service consumer and the Service proxy. What if the proxies move or are unavailable? And how do Service consumers find out about the Service proxy in the first place? The answer to this challenge is the use of registry-based late binding. In this scenario, the Service consumer pings the registry to find out where the proxy is, perhaps based on where the Service consumer is currently located and with what protocols it can communicate. One would think that we can make the Service consumer directly bind to a Service provider after finding its metadata in the registry, but that would be a mistake. The problem is that if the Service provider is using some alternate protocol or is in some location that the Service consumer cant reach, well have a communications problem. So, even in the case where were discovering metadata in a registry, we still dont want direct communication between a Service consumer and provider.
Therefore, the optimal solution here is to use registry-based late-binding to help the Service consumers find proxies and also to help them resolve Service providers to WS-Addresses. Then, the Service consumer communicates with a proxy, providing the WS-Address and required metadata for communication. This allows Service consumers to be quite simple, leveraging a network of distributed, and most likely heterogeneous, Service proxies doing the communication on behalf of Service consumers. In this scenario, companies can implement Service proxies using and mixing a wide range of technologies and approaches including hardware-based network intermediaries, software proxies, and even agents that reside on Service consumer end-points.
Finally, in the above scenario, wheres the ESB? Nowhere, unless you believe that the ESB is a pattern and not a product (as IBM once espoused). Is the ESB needed? Not necessarily. Will it make you feel better to buy one anyways? Maybe. Of course, what are these proxies? Surely ESB vendors will want to have you believe that their products can be used as proxies in this configuration, and thats absolutely true. They certainly can. The point is not that ESB's can't make good proxies. They certainly can, and I want to specifically point that out. But you can also get away with not using one or using your existing infrastructure and it will all still work. The sad truth is that most integration architects are not using ESBs as Service proxies, but rather as EAI 2.0 hub-and-spoke or message-bus brokers that facilitate Web Services Integration, not SOA.
The ZapThink Take
If you think about it, weve already implemented complex, heterogeneous networks in this fashion before. Computers leverage DHCP, gateways, and routers to do networking in a way thats far superior to direct, point-to-point IP communications. And this is basically what were saying about Service communication. Understanding this is a realization that this is not a technology issue, but an architectural one.
To get SOA, you have to get it in your head that the Service concept is an abstraction, not an interface. Once you understand the abstraction, then it will all become that much clearer. So, you need to do some soul-searching with regards to your SOA initiative. Are you doing Web Services Integration (WSI) with EAI 2.0, or have you really changed your architecture to become Service-oriented in an environment of heterogeneity and continuous change? If you are doing WSI on EAI 2.0, you will find yourself continuously solving the same problems. With SOA, you will find yourself able to continuously solve new problems.
- I would support your conclusion, Amit. I am writing the same in the article. Business Operation is the support for the business, nothing more, it is not the business. The SOA service has to address the key business functions, first of all, and Operations as a support only. In many cases, such SOA service capable of replacing the Business Operation.
Thus, the top-down approach has to start in the key business functions and areas. When the business tasks and requirements are understood, it might be recommended to look at available IT solutions, i.e. applications, and identify what can be reused to support found business tasks. At this point you can perform bottom-up approach. The approaches will meet somewhere in between but under real business function umbrella (traditionally, this ended up under business operation umbrella with all following troubles).
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around