Expressions in integrations - “Last time” expression

Introduction

Integrations are not an easy task and sometimes might require lots of coding, adjustments, and ongoing support. In many cases you need to define additional variables to extend the capabilities of the source system and modify the data you have extracted to match your use case.

At ZigiWave, we strive to make every part of your integrations easy and straight forward. This is why ZigiOps provides different ready-to-use expressions that help you create complex data dependencies in a blast.

We use expressions when we want to define additional variables in ZigiOps, based on modifying some data that we have extracted from the source system. These expressions give us the possibility to create things that the source system itself does not provide.

What can we use expressions for?

Expressions are defined from the Source tab in ZigiOps. Here we define a variable, and we have different types of expressions, depending on what we would like to do. Usually the source field is the one we want to apply the expression to.

There are different options that expressions give us.

We can exchange text, extract part of a text, scan for first n characters (we specify a number of characters), we can format date and time, and so on.

These are the different types of expressions that ZigiOps provides.

They give us the freedom to easily define different actions and triggers for our integration, without the need to write any code.

What is Last time?

The Last time expression is the most common one. This is because in the majority of use cases, when we extract information from the source system, we want to avoid duplicate data and collect only the new things that happened after the last data extraction.

For example, if we want to collect alarms that have been triggered by a system, next time we want to skip the ones that we have already collected and start from there. Through the Last time expression we directly say to ZigiOps that we want to collect only new entities or records from a given source.

It is used almost in every integration, because we use it for metrics, events, incidents, tasks, etc. With most data we collect, we want to avoid duplicate records. Maybe the only exception is topology  there we don't have Last time because we usually do a full sync of the topology. In almost every other case it is present.

How does Last time work?

Step 1: We define a variable  we can name it for example lastupdated or lastcreated (the name does not matter).

Step 2: Then we choose the source field. It needs to be a so called time field or a number because what Last time actually does is to save the biggest new value. So it should be a date/time or a numeric field. For example {system.createddate} or {lastcreated}.

Step 3: Once we define the expression, then we can use it in the trigger conditions. For example, we can say: {system.createddate} greater than lastcreated  the expression we just created.

In the expressions section  we define the base on which the expression is calculated, and in the Trigger condition section, we define the condition that will trigger collecting new entities (for example Tasks, Incidents, etc.)

All other expressions, except Last time, are calculated at the Extract level (when we are extracting data).

Last time is the only one that is calculated immediately after Load, and even after Response.

Using Last time as a retry mechanism

When we extract, for example, 5 new tasks, we will try to load them and create 5 new issues in Jira. In case this operation fails, we have the option to choose if we will increase Last time or not. This can be used as a retry mechanism. If we do not increase Last time, next time we will attempt to collect the same 5 tasks.

Our default principle is to retry only when we have network errors. During load if there is a network error, we do a retry, because network errors can be temporary and disappear after some time. So next time when we retry, the error might be gone, and we might succeed.

With other types of errors by default we do not retry. So, when there is a network error we do not increment Last time. With any other type of error, we do increment Last time.

Increment means we save the biggest value that we have collected during the extraction. This way next time we will be looking for bigger values than the last collected one.

When we have a successful load, of course, we also increment Last time, to avoid duplicate records.

Why do not always retry? Because this may lead to a loop in the integration. For example, if we want to update a certain issue in Jira, and we provide the key for example: issue.5, but this issue no longer exists in Jira, we will get an error 404  there is no such issue. No matter how many times we retry, this issue will not appear there, and we will never succeed in updating it. This is why with some errors it is not a good idea to retry.

Here is a little secret to reveal  we are planning to provide our clients the freedom to set and use Last time as a retry mechanism however they find fit. This way you can choose after what errors to retry, and after what errors not to. We want you to have this flexibility to use Last time as a retry mechanism.

Examples

Azure DevOps work items to ServiceNow incidents integration

We have 6 different Last time expressions defined:

• For the work items themselves
• For updating if there are changes to the already transferred work items  3 Last times for comments, attachments and changes
• For backsync updates  2 Last times for changes and notes

This way we collect new work items, as well as updates in both directions.

Nagios metrics to OpsBridge metrics integration

We use Last time to keep the latest metrics we have already collected and not to duplicate them.

Summary

To summarize, the idea of Last time is to keep the biggest value from some set of fields that we have defined, and after that look for new ones (bigger) and load them.

Last time is validated once we see what has happened with the execution  it is not validated when extracted, but after a response. Thats the difference with the other expressions.

Expressions help us define variables and conditions that otherwise would not be accessible to us. This way we can easily set and deploy complex integration use cases in minutes. Tricky data dependencies and specific fields are managed in a straightforward and simple way with ZigiOps.

Even though no-code and easy to work with, ZigiOps is very powerful and provides lots of freedom to deal with complicated integrations. If you have a complex integration case and you are struggling with finding the best solution to solve it, book a demo with us and see how easy it is to make it work with ZigiOps integration platform.