Is Your Integration Project Costing More Than It Should?

Over the years I’ve been directly involved in a great number of IT software development projects at Prolifics. And without exception these have all involved integration of a number of software components – whether it is integrating back-end systems such as SAP, plugging in external services from business partners, or the more relatively straightforward integration of new business systems or processes to in-house business services.

And something I’ve seen all of these projects having to deal with is the fact that software integration introduces some very real dependencies in development and test plans and environments – all of which require a fair amount of attention and a non-trivial amount of coordination and resources which could impact the time and budget estimates.

But what if there was a way to reduce the cost and planning impact of these dependencies?

Monica Luke and I tackle this topic in a new IBM whitepaper that discusses the usage of business service virtualization and testing techniques and tools to remove bottlenecks and reduce costs in building today’s modern, interconnected applications.

Many of our IBM customers will be interested in the discussion on example service architecture as well as mobile and portal domains. We briefly contextualize the value for customers using products such as WebSphere Application Server, WebSphere Enterprise Service Bus, IBM Integration Bus (previously WebSphere Message Broker) and MQ, WebSphere DataPower, IBM Business Process Manager, WebSphere Portal Server and IBM Worklight.

Read our whitepaper here:

Use Service Virtualization to Remove Testing Bottlenecks






How do I get started?
This is a great time to highlight Prolifics' very own IV&V (Independent Verification and Validation) practice. IV&V is an important aspect of our global delivery model, providing customers with end-to-end testing solutions that help to improve productivity and quality while reducing overall cost of all software development activities.

Our team has extensive experience in testing large, interconnected, critical business applications, and is ready to help you reinvent the way you approach your testing – backed by a service framework and testing accelerators that have been field tested and refined for over a decade.

Interested in taking a deeper dive? Connect with me!
Twitter - @greg_hodgkinson
Email - ghodgkinson@prolifics.com
LinkedIn - Greg Hodgkinson

To learn more about Prolifics, visit www.prolifics.com.

Gregory Hodgkinson is the Lifecycle Tools and Methodology Practice Director at Prolifics and an IBM Champion for Rational. Previous to that he was a Founder, Director, and the SOA Lead at 7irene, a visionary software solutions company in the United Kingdom. He has 16 years of experience in software architecture , initially specializing in the field of component-based development (CBD), then moving seamlessly into service-oriented architecture (SOA). His extended area of expertise is the Software Development Lifecycle (SDLC), and he assists Prolifics and IBM customers in adopting agile development processes and SOA methods. He is still very much a practitioner, and has been responsible for service architectures for a number of FTSE 100 companies. He presents on agile SOA process and methods at both IBM (Rational and WebSphere) and other events, has also co-authored a Redbook on SOA solutions, and contributes to DeveloperWorks.

Integrating with Salesforce.com

Salesforce.com (aka Salesforce) is the most commonly used cloud-based Software as a Service platform for Customer Relationship Management (CRM). Recently, we have been involved with a lot of customers who are planning to start migrating to or have made a strategic decision to use Salesforce to manage their customer contacts, track sales orders, streamline their sales processes, etc. In fact, Prolifics itself is a Salesforce customer.

Our own experiences, and those with our customers who use Salesforce, have given us an in-depth understanding of what it takes to ensure that this CRM solution is made universally available within the enterprise – to business processes that require customer information or to portal applications that mash up customer information with data from other systems. This blog entry details three patterns that we have commonly used when integrating with Salesforce and that we believe can be reused – entirely at the design level and to a good extent at the implementation level.

1. Enterprise CRM Services – Every enterprise has defined standards when it comes to their enterprise services and data formats. The reusable enterprise services are exposed via the ESB and all the end applications use these enterprise services to communicate with end systems so that the ESB can provide the common functionality and the governance needed when performing system integration. Using this same concept when working with Salesforce allows enterprises to centralize the Salesforce integration at the ESB layer (the ESB communicates with Salesforce using SOAP/HTTP web services), do transformation, security, etc. at one common place and provide clients – processes, portals or mashups, etc. – with the consistent enterprise-wide data representation to which they are accustomed. The enterprise services provide the generic CRM interface to the clients, so that if the CRM system has to be changed, the hundreds of clients that use the CRM system in the enterprise do not have to change. The IBM WebSphere Enterprise Service Bus is commonly used for implementing this pattern.

2. Publish CRM Data – Another common requirement is to ensure that CRM data is passed from Salesforce to back end systems based upon changes that happen within Salesforce to this information. The ESB provides a service (SOAP/HTTP web service) that gets invoked by Salesforce (with all the relevant information from Salesforce) when data of interest changes in Salesforce. This data is then transformed and passed to the back end systems. The benefits of this approach include a centralized service definition on the ESB, transformation of data, centralized security management, supporting legacy applications that are not web service enabled [but still need CRM data], etc. The IBM WebSphere Enterprise Service Bus or IBM WebSphere Message Broker is commonly used for implementing this pattern. [Note: the reverse flow from end systems to Salesforce is handled similarly, instead of being invoked by Salesforce; the ESB invokes a service on Salesforce.]

3. Bulk Load of CRM data – It is very typical for customers to need the ability to bulk load customer data from their existing homegrown systems into Salesforce. These kinds of requirements are also common when mergers and acquisitions happen and new customer data needs to be loaded. During the bulk load of CRM data, there may also be a need to cleanse the information before loading it into Salesforce. The IBM InfoSphere DataStage, IBM InfoSphere QualityStage, and the IBM InfoSphere Information Server Pack for Salesforce.com together support this pattern that involves the definition of ETL jobs that extract data from different sources, cleanse, map the data to Salesforce format, and load it into Salesforce.

I want to conclude by saying that we at Prolifics believe in “eating our own dog food.” Prolifics has a production implementation of IBM WebSphere Enterprise Service Bus that was built using the pattern described above (Publish CRM Data) that enables the data we have in our Salesforce instance to be published to IBM’s Global Partner Portal based on Siebel.

In my next set of blog entries, I will focus on couple of related topics:

• IBM’s acquisition of Cast Iron Systems and how the solution supports these patterns of integrating with Salesforce


• A set of assets that we are currently working on at Prolifics to help customers jump start their own Salesforce integration initiatives.

Rajiv Ramachandran first joined Prolifics as a Consultant, and is currently the Practice Director for Enterprise Integration. He has 11 years experience in the IT field — 3 of those years at IBM working as a developer at its Object Technology Group and its Component Technology Competency Center in Bangalore. He was then an Architect implementing IBM WebSphere Solutions at Fireman’s Fund Insurance. Currently, he specializes in SOA and IBM’s SOA-related technologies and products. An author at the IBM developerWorks community, Rajiv has been a presenter at IMPACT and IBM's WebSphere Services Technical Conference.

Building an Enterprise Application Integration Solution

Rajiv Ramachandran, Practice Director, Enterprise Integration / Solution Architect

Having an integration infrastructure that connects all enterprise systems to one another and provides seamless and secure access to customers, partners, and employees is the foundation of a successful enterprise. I have been involved with a lot of customers discussing their EAI architectures. More often than not, I have noticed that the approaches considered to implement such an architecture are not complete and do not provide the benefits that may be achieved with a well connected enterprise. In this blog entry, I would like to highlight aspects that need to be considered to build an end-to-end integration solution. (Note: This blog entry will not get into the details on how to implement each of these areas, which would result in me writing a book J.)

1. Connectivity – Avoiding point-to-point connectivity and ensuring that you have loosely coupled systems is key to ensuring that your EAI architecture is flexible and can scale.Use an ESB as the heart of you EAI architecture and ensure that your ESB has support for all major protocols (HTTP, SOAP, JMS, JCA, JDBC, FTP, etc.) and comes with adapters for common enterprise applications like SAP, Oracle, Siebel, PeopleSoft, etc.
   
   
2. Patterns - Build a pattern based integration solution. The following is an excellent paper that outlines some of the common patterns used in the integration space:http://www.ibm.com/developerworks/library/ws-enterpriseconnectivitypatterns/index.html
   
   
3. Data - Data is of great significance when it comes to integration. Different systems have different data formats and there are common items to consider that can help you deal with these differences:




• Define canonical data formats and ensure that you have mapping from application specific formats to canonical formats. Understand the various data formats that exist in your enterprise today and evaluate what it will take to map and manage complex data formats.


• EDI data is common in many enterprises and will require special handling.


• Define a strategy for handling reference data, how lookups can be done against this data and how reference data can be maintained.


• Define rules around both syntactic and semantic validation of messages. Do not over do validation as you will pay a price when it comes to performance. Be judicious in what you want to validate and where.




4. Monitoring - An aspect that is often overlooked when building integration solutions is monitoring. Ensure that you have the right framework(s) in place to monitor your integrations and service components. You will need the ability to monitor every protocol the ESB supports. You also need to couple monitoring with notifications and error handling. You will need a strategy for auditing your messages. Again, as in the case with validations, be judicious in what you want to audit and when. There is a price to pay. Define a centralized service for auditing requirements and ensure that all integration components make use of this service.
   


5. Security - Security is the most critical part of implementing an integration solution. I have to say that most customers realize that security is important but sometimes, because of a lack of expertise in how to correctly secure their solution, the end result is a solution that is way more costly and in fact less secure than desired. Some considerations are:




• Define your security requirements – authentication, authorization, encryption, non-repudiation, etc.


• Decide what aspect(s) can be supported by transport level security and when you need message level security.


• Decide where hardware components can be used to better implement security than software components.


• Use open standard protocols so that you can easily integrate with different systems – both internal and external.





6. Service Oriented Integration – With the adoption of SOA, one of the key architectural models used for integration is to connect to service interfaces that are exposed by various systems. With this architecture, it is also now possible to choose what service / functionality you use dynamically at runtime. Another aspect that SOA has brought to the integration world is a policy-driven approach to integration – that is, data that is being passed to a system or between systems is used to check what policy needs to be applied at runtime to determine what service to use and what actions to perform (auditing, validation, encryption etc.). Integration coupled with SOA introduces another component into the EAI architecture – a registry and repository where services and policies are cataloged and can be used to bring about dynamic, policy-driven behavior.

This confluence of pattern based connectivity, data handling, monitoring, security, and service-oriented integration can provide you with a well-connected enterprise that can respond quickly to changing business needs.

Rajiv Ramachandran first joined Prolifics as a Consultant, and is currently the Practice Director for Enterprise Integration. He has 11 years experience in the IT field — 3 of those years at IBM working as a developer at its Object Technology Group and its Component Technology Competency Center in Bangalore. He was then an Architect implementing IBM WebSphere Solutions at Fireman’s Fund Insurance. Currently, he specializes in SOA and IBM’s SOA-related technologies and products. An author at the IBM developerWorks community, Rajiv has been a presenter at IMPACT and IBM's WebSphere Services Technical Conference.

Integrating WSRR (WebSphere Registry and Repository) and RAM (Rational Asset Manager)

Rajiv Ramachandran, Practice Director, Enterprise Integration / Solution Architect

One of the challenges we are facing as part of a current project that we are working on is to demonstrate the integration of WSRR (WebSphere Registry and Repository) and RAM (Rational Asset Manager).

First and foremost there are a lot of questions on the roles and responsibilities of both these products.

• Do we need both products?
• If so, what do we use RAM for and what do we use WSRR for?
• What is an asset?
• Is a service an asset? Can there be other assets other than services? Does that mean only service assets go into WSRR?
• What is a service? Is it just a WSDL? What about the service code? Should that also go into WSRR?
• How do these products work with one another?
• When I make a change in RAM does it reflect in WSRR and vice versa?

There are more questions than answers. I don't have answers to all these questions and as we work on this project, I am sure we will learn more. The team will post more answers here on this BLOG.

Today we took the first step. How can we get WSRR and RAM connected? I should tell you to my surprise I found that it was not difficult to accomplish this.

We had a RAM instance (v7.1) running on a VM Image.

I installed WSRR v6.2 on the same image and since it was for a POC, I used the Derby DB for WSRR. (I had installed WSRR with a DB2 backend and it had worked fine.)

There were just 2 things I needed to do to get the connectivity working

• RAM runs on a WAS instance. WSRR was running on a separate WAS instance. The communication between the two was over HTTPS. So I had to ensure that WSRR server certs were part of the trust store of the WAS server running RAM. Nothing RAM / WSRR specific here, pure WebSphere stuff.
• Log into the RAM WAS console using the following URL (https://localhost:13043/ibm/console/logon.jsp)
• Go to Security -> SSL certificate and key management > Trust managers > Key stores and certificates > NodeDefaultTrustStore > Signer certificates
• Do Retrieve from Port, Provide the following values: Host -> localhost, Port 9443, Alias WSRR
• Log out

We needed to set the configuration in the RAM Admin Console to connect to WSRR.

• Log into the RAM UI as admin
• Go to the admin tab, click on 'Community Name'
• Go to Connections
• Add a WSRR Connection
• Call in LOCAL WSRR
  - The URL to provide is https://localhost:9443/
  - User id and password is admin / admin
  - Do 'Test Connection' and that should be successful
  - Click on Synchronize and you should see the message -> WSRR-Asset Manager synchronization started
  

That’s it. Now I was able to search in RAM and have visibility into assets that were in WSRR. This was a simple test. These were stand alone instances . I assume when we do have clustered configurations we may have some challenges but it was a pleasant surprise to see something work the first time you try it.

Rajiv Ramachandran first joined Prolifics as a Consultant, and is currently the Practice Director for Enterprise Integration. He has 11 years experience in the IT field — 3 of those years at IBM working as a developer at its Object Technology Group and its Component Technology Competency Center in Bangalore. He was then an Architect implementing IBM WebSphere Solutions at Fireman’s Fund Insurance. Currently, he specializes in SOA and IBM’s SOA-related technologies and products. An author at the IBM developerWorks community, Rajiv has been a presenter at IMPACT and IBM's WebSphere Services Technical Conference.

Service Gateway Pattern in WebSphere Enterprise Service Bus (WESB) 6.2

Rajiv Ramachandran, Practice Director, Enterprise Integration / Solution Architect

On of the new patterns that have been introduced in v6.2 of WESB is the Service Gateway Pattern. There are two implementations of this pattern – A Static Gateway and A Dynamic Gateway. In this blog entry I will focus on my experiences on using the Dynamic Service Gateway pattern.

One of the basic requirements that I expect in an ESB is the ability to introduce an ESB in the middle between a client and a service (without impacting either the client or the service) and be able to do provide value added services like logging, monitoring, auditing etc. This is exactly what we are able to accomplish with the Dynamic Service Gateway pattern. This gateway pattern is implemented as a mediation component. The interface for this mediation component has 2 operations, a request operation and a request response operation. Both these operations accept xsd:anyType (they are not bound to the type definitions exposed by any particular service). The reference partner for this mediation also maps to the same interface. The reason this pattern is dynamic, is because there is no coupling to services at compile time. You can dynamically set the service endpoint at runtime and invoke that service.

However what needs to be understood is that this pattern implementation works only for SOAP 1.2 based JAX WS Web Services. The reason why this restriction exists is because in JAX-RPC the invocation was RPC based -> a particular web service operation had to be specifically invoked. In JAX-WS the invocation is more messaging based. There is no specific operation invocation required. The following article series provide an in depth analysis of the differences between JAX-RPC and JAX-WS.
http://www.ibm.com/developerworks/library/ws-tip-jaxwsrpc4/index.html

The following diagrams show a sample implementation of this pattern for monitoring and auditing service invocations.


Assembly Diagram

Mediation Flow

Look up Table

Audit Table

Rajiv Ramachandran first joined Prolifics as a Consultant, and is currently the Practice Director for Enterprise Integration. He has 11 years experience in the IT field — 3 of those years at IBM working as a developer at its Object Technology Group and its Component Technology Competency Center in Bangalore. He was then an Architect implementing IBM WebSphere Solutions at Fireman’s Fund Insurance. Currently, he specializes in SOA and IBM’s SOA-related technologies and products. An author at the IBM developerWorks community, Rajiv has been a presenter at IMPACT and IBM's WebSphere Services Technical Conference.

Dynamic Routing using WebSphere Enterprise Service Bus (WESB)

Rajiv Ramachandran, Practice Director, Enterprise Integration / Solution Architect

One of the most common requirements I have heard from customers who are using an ESB is to say that I want to dynamically route information to several of my end systems. I don't want to change my code when i introduce a new system to the mix that requires the same data.

Careful analysis is required to implement a solution to this requirement as there are two parts to this.

1. The ability to invoke an end point dynamically (the requirement statement is crystal clear about that).
2. The second part is how do I define my routing logic? How do i describe it? How can i change it at runtime?

Setting the endpoint header dynamically on the endpoint is supported by the Service Component Architecture (SCA) framework and therefore we do have an easy solution to the first part of the requirement. However there is no standard primitive or a defined approach in the WESB product to achieve the second aspect of the dynamicity requirement.

There are multiple solutions to this

1. We can use WebSphere Business Fabric (WBSF) to provide this capability.
2. Or we can build custom solutions to achieve this goal.

One such solution approach that we have used is to use Business Rules component in WebSphere Process Server (WPS) to be a 'Routing Controller' and define business rules to describe the routing logic. The out of the box capability of business rules in WPS provides us with an editor to edit to these rules and dynamically update them at runtime. The pros of this approach are that we don't have to custom code any of the logic and we are able to use the out of the box rules component. However, since the business rules components are part of WPS, this solution is not applicable for pure WESB customers.

Rajiv Ramachandran first joined Prolifics as a Consultant, and is currently the Practice Director for Enterprise Integration. He has 11 years experience in the IT field — 3 of those years at IBM working as a developer at its Object Technology Group and its Component Technology Competency Center in Bangalore. He was then an Architect implementing IBM WebSphere Solutions at Fireman’s Fund Insurance. Currently, he specializes in SOA and IBM’s SOA-related technologies and products. An author at the IBM developerWorks community, Rajiv has been a presenter at IMPACT and IBM's WebSphere Services Technical Conference.

SOA and Off-Shore Project Models

Jonathan Machules, Technology Director

SOA is, as it was hyped by the marketing machines years ago, the next maturity level in Architecture. Service Orientation adds another level of design, development and maintenance considerations to projects. Another trend in the industry is the off-shore model of development of these projects. You can see why project success with these maturing trends is largely dependent on the personnel involved.

At Prolifics we are leaders in both these areas. We have expertise in SOA Methodology, Design and Architecture across all functional and non-functional domains complimented by a proven off-shore methodology that delivers.

On a recent SOA project using off-shore resources I had 2 months to Architect and Design an Enterprise Integration Platform for a large financial company seeking to use a combination on-shore/off-shore model for the Design and Development of Enterprise Integration system based on SOA.

This combined model works well with the right pieces in personnel in place. Capable developers with the ability to learn new technologies, strong project management and architectural leadership. Couple this with proven communication strategies and process for corresponding tactical direction and deliverables and there is a model that can work in the face of new technologies and regionally separated personnel.

The pay-off for this model combination is reduced cost and time to market. Given the economic times I foresee we will see a fair amount of interest and subsequent implementations with such models.

-Jon

Jonathan Machules first joined Prolifics as a Consultant, and is currently a Technology Director specializing in SOA, BPM, UML and IBM's SOA-related technologies. He has 12 years experience in the IT field — 2 of those years at Oracle as a Support Analyst and 10 years in Consulting. Jon is a certified IBM SOA Solution Designer, Solutions Developer, Systems Administrator and Systems Expert. Recent speaking engagements include IMPACT on SOA End-to-End Integration in 2007 and 2008, and SOA World Conference on SOA and WebSphere Process Server in 2007.