Systems Integration | Headpsring

Integrating your systems with a third-party platform may be the fastest way to add new functionality to your application—but the path to productivity isn’t always smooth. Services that your users need may not always be responsive or available, so are you taking the best steps to ensure that the experience is not compromised? Systems integration hiccups—like pesky API outages—can lead flighty customers to your competitors, while poor-performing services cause headaches and churn in your IT Department. Your ability to innovate and drive new features will be stunted as developers chase down issues. Ultimately, ill-planned systems integrations will affect your bottom line. The only way to avoid risks to the business caused by integration glitches is to consider each possible scenario (we’ll help you below), and devise a robust plan for supporting a strong enterprise architecture,

Finding the gaps

We recently created a custom data entry application for a client that would post data to an ERP’s API. The client team and the ERP’s specialists worked together and did everything by the books. They knew what data they needed to collect and where the ERP would store their data and even how it would be formatted. However, they were missing one key piece in this conversation: They never discussed in detail how the data was going to get from the custom app to the ERP. It was assumed that the systems would just seamlessly “talk” to each other without much fuss.

The assumption that everything would just work created large architectural gaps. The implementation would require the custom app to perform close to a dozen sequential API calls between the user clicking “Submit” and loading the next page. Yet, the ERP did not provide us the ability to recover from any errors that could occur along the way, like locking POs or rolling back transactions. The new application could potentially create dozens of broken or duplicate POs in the ERP per day. This was a disaster waiting to happen, but was easily mitigated with the right plan.

Everything must break

No system is safe. They all break eventually. You must understand how your application will behave when system interactions with a third party go bad. Ask the right questions so that you can design a robust, resilient, and flexible system—It should not behave erratically when it ventures off the happy path.

Consider the following examples:

  • What if the interaction fails because of the network?
  • What if the interaction succeeds at the third party, but we don’t receive a response?
  • What if the third party returns an unexpected response?
  • What if the third party is down for a few minutes? Hours? Days?
  • How does the third party handle concurrency and idempotency issues?
  • How do we control race conditions?

These discussions should take place around every systems integration your application makes. All systems are unique, so the answers for each, and your plan for each, will be different. It is important to get creative, since goofy things will happen in the lifetime of your application.

The systems integration game plan

You must understand why each interaction is important to your business. Different integrations need different failure-handling depending on why they are important. At one end of the spectrum, your landing page fails to fetch your latest tweets, and at the other, you fail to process a payment. These have drastically different business implications, so you need to adjust your approach accordingly.

Ask your team:

  • Is it critical that this service call executes in a certain time? Seconds? Minutes? Hours?
  • Is an end-user expecting feedback on the success or failure of this call?
  • Are any of your other business processes dependent on this call’s success? How do we notify those processes about failure?
  • If a service is down, how will this affect our other business processes?
  • Can we ignore failures? Should we retry them? How many times?
  • Do we need to have staff in place to track the success or failure of these interactions?

It is important to understand the tools you have available when building your game plan. Modern applications have the ability to retry, queue, and cache calls in a variety of ways. Headspring’s Chief Architect, Jimmy Bogard has an excellent video on designing resilient distributed systems, which provides useful insights on the many of the ways you can plan for failures in order to secure success.  

Bridging the gaps

In our ERP integration scenario, we had a couple options to pursue. We could put in place a messaging system that coordinated the various API calls that we needed to make. This messaging system would attempt to prevent PO duplication, and escalate if it failed after several retries. Another option was to build it as part of our app, and the ERP team would not have to change anything on their side. But this solution involved a lot of extra work and decreased the code maintainability. There was also still no way to roll back changes. Finally, there was not enough time to get it in place before the deadline.

In order to create a strong solution while keeping the project on track, we worked with the ERP to automate the intermediate tasks. Our application would make one API call and the data would be processed on the ERP system. They could lock records to prevent race conditions and roll back changes if necessary. The execution was quick because we didn’t have to wait for a dozen or so extra API calls. The custom API would send good or bad responses depending on the outcome. If we got a failure or bad response, we would fail-fast to the user instead of attempting to retry. This rapid solution resulted in a more robust system on both sides, and a better end-user experience.

Think the worst, plan for the best

Don’t ever assume the worst will never happen, because it will. It’s necessary to be pessimistic when planning your integration, particularly one that’s integrated with a third party platform that you don’t have full purview over. Great system architects are natural skeptics with endless fountains of “What If?” questions. Challenge third parties with the hard questions at the outset—doing so could save your project, timeline, and most of all, your budget.

Let's Talk