How to prepare for a systems integration: governing the data exchange among cloud IT applications
With the advent of SaaS applications, companies have been able to take advantage of quick implementations and a short time to value. However, exchanging data among a new SaaS application with other SaaS and on-premise applications is sometimes an afterthought. This is especially common when a business unit decides to use a cloud software application to solve an urgent business problem without a representative from IT involved.
Connecting IT systems properly for the exchange of information is critical for smooth business processes. That’s why it’s so important for system integrations to be considered as part of overall corporate IT architecture, and from the very beginning of software implementation projects.
These considerations should not be limited to designing interfaces. Stakeholders need to plan for how to handle potential operational issues, such as:
- How are integration issues detected if they occur?
- How will they be addressed and by whom?
This type of strategy is known as integration governance.
Why integration governance is important
During a SaaS implementation, a lot of information may be exchanged among applications. And with a large number of information protocols to be managed, it’s crucial to ensure data integrity.
Each business system may be compliant with a data access policy but creating an enterprise governance plan to ensure that all business systems speak to each other seamlessly without loss of data integrity is often a challenge. Many enterprises monitor integration points manually, which may not be effective, especially with higher data volumes.
There are a handful of challenges that we frequently see organizations struggling with when it comes to system integrations, including:
- Dependency on external factors such as network connectivity or company-specific security policies
- Inability to set up integrations based on industry standards or SaaS provider recommendations
- Lack of communication and/or clear accountability to address exceptions
- A business process or underlying application issues being reported as integration issues
In this blog, we’ll review some of the basic principles of systems integration.
Three integration methods – and when you should (or shouldn’t) use them
There are three main integration methods that you can choose from.
- Web Service Push – SaaS pushes data consuming an external web service API
- Event-based data transmission provides an almost real-time data exchange
- The system of record has better control, where any failure is reported immediately
- The system consuming the data needs to publish an API
- Web Service Pull – external system pulls data consuming the SaaS web service API
- Works well when data is not impacting real-time business, e.g. data is required for reporting purposes
- The system consuming the data does not need to publish an API
- The system of record does not have control over data transmission
- Data transmission does not occur in real-time
- Extra effort is needed for handshaking of data and confirming receipt
- File-Based – files are exchanged via Secure File Transfer Protocol (SFTP)
- Easy implementation compared to a web service-based integration
- Less security setup is required
- Data cannot be transmitted in real time
- It requires a more complex process for exception handling due to its offline processing
Looking at the pros and cons of each, web service push is our preferred integration method as it provides almost real-time integration and better exception handling.
Once integrations have been set up and data starts flowing between applications, what happens if something goes wrong?
When you have data integration errors or failures, the information flow is impaired and your business processes may be disrupted. Your organization needs to be prepared to deal with these exceptions. This includes how do you detect an exception, and based on the nature of the integration issue, who will be on point to address it.
System integration exceptions can be grouped into three general categories. Let’s review these categories and ways to manage them.
- Data validation exceptions
- Business validation exceptions
- System exceptions
A data validation exception occurs when any mandatory field is not available in the transaction. Mandatory field requirements must be aligned between the source and target systems as well as defined and documented in the design phase.
When an issue occurs, the system that is processing the data must be notified so that it can be fixed in the system(s) of record and will not occur again.
This type of exception is meant to be addressed by either a data steward or a business user.
A business validation exception occurs when a business rule is violated during a transaction or when data required to complete the transaction is not set up in the target system. To function properly, business rules must be defined and documented in the design phase.
If issues arise, they can be corrected by adjusting the data setup in the system that is processing the data.
Just like with data validation exceptions, this type of exception is meant to be addressed by either a data steward or a business user.
A system exception occurs due to technical issues, such as a network failure, missing authentication, or a connection time out. These types of exceptions should be addressed by the respective SaaS provider(s)’ customer support processes.
SaaS applications usually trigger email notifications with integration error messages. Make sure to nominate an individual like a data steward or system administrator who receives such messages and is trained to resolve exceptions or to communicate with the support team of your SaaS provider.
Managing multiple applications from different software providers requires preparedness on the user side.
Complex business processes may require integrated data flows among multiple applications. This requires careful design and planning of how to exchange data, how to communicate among stakeholders, how and when to maintain and upgrade different systems, and much more. Managing your application landscape may be handled in-house or delegated to an external service provider.
A critical component to successfully governing a SaaS integration architecture, systems integration setup, and ongoing management is the role of a data steward. A data steward is responsible for defining and implementing policies and procedures for the day-to-day operational and administrative management of systems and data. This includes the intake, storage, processing, and transmission of data to internal and external systems as well as handling exceptions.
The lesson here is: be prepared. Think about this early in your implementation project. Pick the best integration method and assign a data steward or system administrator who can analyze and resolve any integration failures that may arise.