Loading ...
Sorry, an error occurred while loading the content.

[ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate

Expand Messages
  • Gervas Douglas
    [ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate [ZapFlash] Why Service Consumers and Service Providers Should Never
    Message 1 of 84 , Feb 29 4:24 AM
    • 0 Attachment
      [ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate[ZapFlash] Why Service Consumers and Service Providers Should Never Directly Communicate

      Why Service Consumers and Service Providers Should Never Directly Communicate

      Document ID: ZAPFLASH-2008228 | Document Type: ZapFlash
      By: Ronald Schmelzer
      Posted: Feb. 28, 2008

      Edward R. Murrow once said, “People say conversation is a lost art; how often I have wished it were.” How true it is in the vendor and media-spun world of IT and even more so in the case of SOA. Many people think they are doing or talking about Service-Oriented Architecture (SOA), but most of the time they’re really doing Web Services Integration (WSI), developing Just a Bunch of Web Services (JBOWS), or even just point-to-point XML (what many would like to call REST, but even in most cases is not even truly Representational State Transfer [REST], which is also an architectural form). Even after 8 years of evangelism, research, advisory, and deep consulting, we’re still finding a large percentage of so-called enterprise architects pursuing SOA projects as if they were point-to-point integration projects.

      We’ve discussed many times from an economic, technical, architectural, and philosophical perspective the problem with not understanding that SOA is an architectural concept, so I won’t bore or nag you further in understanding that SOA is not just another name for integration. However, we need to crack this nut and crack it definitively if this industry is to make progress from its current inefficient, ineffective, unpredictable, overly-expensive, overly-complex, and rudimentary form to the predictable, reliable, cost-effective, disciplined, and easily measured part of the business organization that we know it should be.

      While we’ve made our pleas to the business, architectural, and process-driven parts of the organization and to vendors in previous ZapFlashes, we want to focus this ZapFlash on where the problem specifically lies: within the IT organization, and specifically the part that concerns themselves with so-called integration issues. Folks (and I’m speaking directly to you integration-centric techies), you are getting in the way of making SOA a success probably due to your own misunderstandings and misconceptions of how SOA should work. And to correct your misunderstanding, I need to speak your language: the language of integration technology.

      The Problem of Service Consumer to Service Provider Communication
      In typical integration environments, you’re primarily concerned with the movement of data or the amalgamation of business logic to achieve a specific task. In that vein, most integration developers think about inputs (“gazintas”), outputs (“gazattas”), transfer, and transformation. That is, they think about how they’re going to get data or application functionality out of one system, how they’ll get it into another system, the mechanism they’ll use to get that information or functionality from one place to another, and then what they need to do with it while it’s en-route to transform it from one format, protocol, context, process, or policy into another. So, while architects are chemists – that is, scientists who are trying to understand a system and how it can all be put together, integration developers are chemical engineers – focusing on just the mechanisms of transformation.

      >From this integration-centric mentality, most integration

      developers see the concept of the Service as just another means of gazinta or gazatta. That means these integration-heads need to find something else to do transfer and transformation. Vendors to the rescue! Most vendors responded to this desire by simply dusting off their existing integration middleware. End-users responded easily to this insignificant change and voila, now they have transfer and transform that can deal with their Service-based gazintas/gazattas. But does this make SOA? Oh no, no, no. But we’ve already beaten you on the head with that a million times. So, let’s take a different tack. The reason why this approach is wrong is because Services Consumers (what you might think of as a gazinta) should never communicate directly to Service Providers (what you might erroneously think is a gazatta).

      Why? First, the main concept of SOA is that we want to deal with frequent and unpredictable change by constructing an architectural model, discipline, and abstraction that loosely-couples the providers of capability from the consumers of capability in an environment of continuous heterogeneity. This means that we have to somehow develop capabilities without the knowledge of how they might be used. As we delved into the Seven Aspects of Loose Coupling in a previous ZapFlash, this means that we need to have significant variability at a variety of aspects including the implementation, infrastructure, contract, process, policy, data schema, and semantic aspects. Having this variability allows companies to have stability in their business and IT even though the IT and business continue to change. Agility, predictability, reliability, visibility, and cost-effectiveness all become that much more realistic when we can achieve that sort of abstraction.

      But to achieve that abstraction means that we can’t afford things to break when things change. Lots of things change in an IT or business environment: location and availability of services, versioning of Service implementations and contracts, changes to business processes and policies, new or changing requirements for reporting, visibility, and control, and of course the need for new capabilities to solve emerging business problems or capitalize on new business opportunities. However, if you think that a Service is just an API, then all hell breaks loose when you try to connect a Service Consumer directly to a Service Provider. What if the provider moves? What if the Service contract changes? What if the provider is no longer available? What if you now need a new communication mechanism or change to a data schema?

      The first knee-jerk reaction to this complexity by most integration “architects” is to put some piece of technology between the Service consumers and providers to ameliorate this problem. The impulse is correct: you can’t have direct communication if you want to have variability. However, the execution is all wrong. If you are relying on black-box, and often proprietary, technology to do the heavy lifting of managing Service communication differences you’ll simply shift all the complexity and work from the end-points (the Services) to an increasingly more complex, expensive, and brittle middle point. Welcome to EAI 2.0, the secret and unpublicized real name for most so-called Enterprise Service Bus (ESB) products – or at least the way that many are using them.

      (Re)Introducing the Service Proxy
      So, if reworked middleware isn’t the solution, what is? Well, of course the answer is architecture. But that’s too pat of an answer, so to speak again in the parlance of integration developers, let’s explore how a system would be designed if you can’t count on any particular vendor’s solution to be in the middle to make it all work.

      Let’s say you want to build a Service Consumer to consume or compose some functionality, but you don’t know where that functionality is or even how to communicate with it. There are at least three ways of handling this particular challenge. First, you can talk to a proxy (or broker or intermediary) in a language that you do understand and that’s in a location that you already know and send a request to this proxy that will act on the behalf of the Consumer. This is the “envelope within an envelope” concept. Basically, you enclose a message to deliver to someone whom you know, but don’t know where they are, within an envelope that is addressed to a party that you do know. That party then opens the envelope and addresses the internal envelope to the end recipient based on a look-up of some sort. This approach simplifies the problem of the Service consumer having to know about changes since they just need to know about the proxy and the end-recipient. From a technology perspective, the use of WS-Addressing simplifies this approach. In essence, a Service consumer just needs to know the WS-Address of a Provider and hand that to a Service proxy to resolve and deliver.

      We’ve discussed this concept of the Service Intermediary before. The problem with this approach is that the Service proxy still needs to find a way to deliver the message, and if the proxy contains all the rules internally in the black-box, we have the same problem we have with EAI 2.0-style ESB solutions. This is where the registry steps in. Service proxies should not store their own rules, but rather get all their routing, location, and policy-based binding rules from a central system of record. In SOA, this is the role of the registry. Proxies can cache the results of registry look-ups for performance enhancement, but the metadata themselves should come from the registry.

      So, registry-configured Service proxies solve many of the problems of Service communication, but it still leaves open the problem of the connection between the Service consumer and the Service proxy. What if the proxies move or are unavailable? And how do Service consumers find out about the Service proxy in the first place? The answer to this challenge is the use of registry-based “late binding”. In this scenario, the Service consumer pings the registry to find out where the proxy is, perhaps based on where the Service consumer is currently located and with what protocols it can communicate. One would think that we can make the Service consumer directly bind to a Service provider after finding its metadata in the registry, but that would be a mistake. The problem is that if the Service provider is using some alternate protocol or is in some location that the Service consumer can’t reach, we’ll have a communications problem. So, even in the case where we’re discovering metadata in a registry, we still don’t want direct communication between a Service consumer and provider.

      Therefore, the optimal solution here is to use registry-based “late-binding” to help the Service consumers find proxies and also to help them resolve Service providers to WS-Addresses. Then, the Service consumer communicates with a proxy, providing the WS-Address and required metadata for communication. This allows Service consumers to be quite simple, leveraging a network of distributed, and most likely heterogeneous, Service proxies doing the communication on behalf of Service consumers. In this scenario, companies can implement Service proxies using and mixing a wide range of technologies and approaches including hardware-based network intermediaries, software proxies, and even agents that reside on Service consumer end-points.

      Finally, in the above scenario, where’s the ESB? Nowhere, unless you believe that the ESB is a pattern and not a product (as IBM once espoused). Is the ESB needed? Not necessarily. Will it make you feel better to buy one anyways? Maybe. Of course, what are these proxies? Surely ESB vendors will want to have you believe that their products can be used as proxies in this configuration, and that’s absolutely true. They certainly can. The point is not that ESB's can't make good proxies. They certainly can, and I want to specifically point that out. But you can also get away with not using one or using your existing infrastructure and it will all still work. The sad truth is that most integration architects are not using ESBs as Service proxies, but rather as EAI 2.0 hub-and-spoke or message-bus brokers that facilitate Web Services Integration, not SOA.

      The ZapThink Take
      If you think about it, we’ve already implemented complex, heterogeneous networks in this fashion before. Computers leverage DHCP, gateways, and routers to do networking in a way that’s far superior to direct, point-to-point IP communications. And this is basically what we’re saying about Service communication. Understanding this is a realization that this is not a technology issue, but an architectural one.

      To get SOA, you have to get it in your head that the Service concept is an abstraction, not an interface. Once you understand the abstraction, then it will all become that much clearer. So, you need to do some soul-searching with regards to your SOA initiative. Are you doing Web Services Integration (WSI) with EAI 2.0, or have you really changed your architecture to become Service-oriented in an environment of heterogeneity and continuous change? If you are doing WSI on EAI 2.0, you will find yourself continuously solving the same problems. With SOA, you will find yourself able to continuously solve new problems.

    • Michael Poulin
      I would support your conclusion, Amit. I am writing the same in the article. Business Operation is the support for the business, nothing more, it is not the
      Message 84 of 84 , Apr 12, 2008
      • 0 Attachment
        I would support your conclusion, Amit. I am writing the same in the article. Business Operation is the support for the business, nothing more, it is not the business. The SOA service has to address the key business functions, first of all, and Operations as a support only. In many cases, such SOA service capable of replacing the Business Operation.

        Thus, the top-down approach  has to start in the key business functions and areas. When the business tasks and requirements are understood, it might be recommended to look at available IT solutions, i.e. applications, and identify what can be reused to support found business tasks. At this point you can perform bottom-up approach. The approaches will meet somewhere in between but under real business function umbrella (traditionally, this ended up under business operation umbrella with all following troubles).

        - Michael

        ----- Original Message ----
        From: "Karandikar,Amit" <amit.karandikar@...>
        To: service-orientated-architecture@yahoogroups.com
        Sent: Friday, April 11, 2008 4:16:42 PM
        Subject: RE: [service-orientated-architecture] Level 0 architecture - questions

        Thanks Steve, Michael for your guidance. I'll take a look at the material that you guys are referencing. ...maybe it will help me convince management that a top-down approach is the way to go.
        I think I have some consensus within IT about what Level 0 should look like - but the debate was over whether the operations area of the company was to be considered at that top level. Would operations really be more of a support service? While modeling level 0, I found it easier to determine key business areas like key use cases of an enterprise application. So taking the actors of the enterprise, the key business functions (a use case to me) became the different key business areas. In that regard, operations fit in. Such an analogy may still be applicable at other levels also(Level 1, 2), but I thought it got us a more consistency on decisions. Any thoughts on this approach?

        From: service-orientated- architecture@ yahoogroups. com [mailto:service- orientated- architecture@ yahoogroups. com] On Behalf Of Steve Jones
        Sent: Friday, April 11, 2008 3:22 AM
        To: service-orientated- architecture@ yahoogroups. com
        Subject: Re: [service-orientated -architecture] Level 0 architecture - questions

        I'm talking about BOM in the concept of value networks http://service- architecture. blogspot. com/2008/ 03/networked- economies- require-services .html so the core operating model rather than the current operating procedures.

        In this sort of approach there isn't really much difference between the BOM and a BSA, but there is a big difference between this type of BOM and "traditional" BOM models.


        On 10/04/2008, Michael Poulin <m3poulin@yahoo. com> wrote:

        I have concluded recently that Business Operating Model  is a misleading thing. Business Operation is a disposable and vulnerable part of business and one of the major problems of IT is its constant intention to follow this ever changeable business function implementation.

        From the business model perspective it does not matter how business operates until it provides for the function success. I think SOA would be much more successful if it starts to address business model, i.e. business services, processes and, in the lower level, business functions and features. In this case IT would be able to perform as Business Operation extending its area of covering wider and wider.

        - Michael

        ----- Original Message ----
        From: Steve Jones <jones.steveg@ gmail.com>
        To: service-orientated- architecture@ yahoogroups. com
        Sent: Thursday, April 10, 2008 11:43:28 AM
        Subject: Re: [service-orientated -architecture] Level 0 architecture - questions

        On 09/04/2008, Karandikar,Amit <amit.karandikar@ edwardjones. com> wrote:
        > Hello
        > I am looking to start up an effort in the company to start creating the high level service
        > architecture that may potentially help drive the 10year road map for IT in the organization.

        My first hint would be don't drive a 10 year roadmap, 5 years out is
        about as far as you should be planning it needs to be purely
        aspirational beyond that. Lots of companies in 1998 for instance
        would have seen the internet as a "new" channel and only a small
        element and opportunity so its brave to commit to 10 years out.

        I'm not a huge fan of IT strategies, except where they are about
        implementing the business strategy. Saying that you are trying to
        understand the business strategy to understand the IT place in it (the
        truth) often helps open up the discussions.

        > I've
        > been going through a lot of material in trying to get a sense of what would be the best way of
        > going about doing this and Steve's book on Enterprise SOA adoption Strategies definitely has
        > helped. However, there are a couple of problems.
        > Management does not believe this can be done. So involving people from the business areas
        > all at once is out of question.

        One thing to read up on is "value networks" which share alot of the
        same principles of the BSA stuff. I've had some success recently by
        saying that we are creating a Business Operating Model which combines
        the value networks (biz view) with the business services (biz view).
        If that doesn't work then....

        > Which mean, I will really need to tip-toe around the organization to get the information to put
        > up this "big picture" - unfortunately another IT driven effort.
        > I have plenty of experience on Service projects (small and fairly big), however, on bigger
        > efforts, I had more interaction with the business folks and the approach was still driven from a
        > project focus (definitely helps on the scope side of things).

        Projects help hugely, and my experience has been that if you have a
        number of service architectures from projects and then look to connect
        them with a more business centric piece then people begin to
        understand what you are after.

        > Given my situation, what approach do I take? For defining capabilities, where do I begin? For
        > a corporation with so many functions, looking at the functions from the 70,000ft level is indeed
        > an arduous task. The worry is that the functions will more come close to the
        > departments themselves than modeling the various functions that the organization actually
        > performs.

        The average manufacturing company has (IME) four Level 1 services -
        Manufacturing, Sales, Finance, Logistics/Distribut ion

        If you then look in Manufacturing then you have between 6 and 8 (from

        The key isn't to try and model the weeds in one go its to look at the
        structure. In terms of the departmental element just think about what
        is done rather than the structure (I've been in places where Marketing
        and Logistics were together because the Head of Logistics kept the
        marketing department, his previous role, because he liked the

        > Also, is there any way to model the competitive advantage of a corporation while determining
        > the functions - or is that implicit in the discovery?

        The heatmapping exercise in the book covers exactly that in that it
        focuses on which services are important and what sort of measurement
        and cost structure should be used for the business case and contracts.
        This is the bit that most business people have got excited about,
        especially in companies that are looking to outsource.

        > Have many more questions, but answers to these will getting me some feel of where I need to
        > take this. My drive here is to avoid IT asking for tools like an ESB as another way to get SOA
        > implemented - we all know that's not going help solve the bigger issue.
        > Any help or assistance would be appreciated.
        > Sincerely
        > Amit Karandikar
        > Application Architect
        > If you are not the intended recipient of this message (including attachments) , or if you have received this message in error, immediately notify us and delete it and any attachments. If you no longer wish to receive e-mail from Edward Jones, please send this request to messages@edwardjone s.com. You must include the e-mail address that you wish not to receive e-mail communications. For important additional information related to this e-mail, visit www.edwardjones. com/US_email_ disclosure

        ____________ _________ _________ _________ _________ __
        Do You Yahoo!?
        Tired of spam? Yahoo! Mail has the best spam protection around
        http://mail. yahoo.com

        Do You Yahoo!?
        Tired of spam? Yahoo! Mail has the best spam protection around
      Your message has been successfully submitted and would be delivered to recipients shortly.