Field Mapping in integrations – explained

Extract, Transform and Load

To have an integration, we need to have transfer of data between two systems. This happens through the so called ETL  Extract, Transform and Load. These are the three main processes of each integration. We extract the data from the first system, we modify it, and then we send it to the other system.

The middle phase  Transform  is usually more technical and translates the data from the first system, so that it can be properly understood by the second system. For example, if one of the systems works with JSON, and other with XML, or a different language, the data needs to be modified in order to fit in the target system.

In ZigiOps all these transformations, are done in the background. The client does not need to care about them, no adjustments or modifications are necessary. This is why our UI is mainly focused on the Extract and Load phases.

ETL starts with extracting the data from the source system. This is quite important as extracting data correctly sets the stage for the success of the following processes.

The load is the last step of the ETL process and involves importing extracted and transformed data into a target database or system.

In ZigiOps, once we have defined the systems and the entities (type of data) that we want to transfer, then at the operation level  in every Action we have a Source tab and a Field Map tab. The Source tab refers to the Extract phase (data collection) and the Field Map refers to the Load phase (data delivery).

What is field mapping?

The mapping, as a component of integrations, defines how we will send data to our target system.

In the Source tab we define what data we will collect and how we will collect it. For example, if we have defined that the entity will be task  which tasks we want to collect  just the newest, only the ones that are assigned to a specific user, etc. The filters and conditions in the Source tab are about what data we collect.

The Field Mapping, on the other hand, defines what do we want to send to the entity in the target system. Which fields in the target do we want to create or update, and what values we want to populate or assign to them. These values usually come from the source system.

For example, if we want to create an incident in ServiceNow from a task in Jira, the mapping allows us to define we want an incident in ServiceNow, where its short description field will contain the value from the Jira Summary filed.

How field mapping works?

In the mapping we have the following options:

• Direct mapping from a field of the source system to a filed in the target system  this is the actual transfer of data

• Hard coded value  usually this is when ZigiOps covers some requirements, so that the use case can be completed successfully
• A combination of the two

Lets take again the example where we are transferring Jira tasks to ServiceNow incidents, and we are mapping the incident description. In case the value is hard-coded, every incident that we create in ServiceNow will have the same short description. In this case it does not come from Jira, but from a value in ZigiOps. This is called free text string hard coded option. Some fields require this. Usually it is used when we have a required field in one of the two systems, that is not required or is completely missing in the other system.

An example is BMC Remedy integration with Jira. If we are transferring Jira tasks to Remedy incidents, Remedy has required fields that do not exist in Jira. For example, Remedy requires First name and Last name of the user, and if we want to create an incident there, we need to have some values in these fields. In Jira this user might not exist, and these fields are not present in Jira at all.

In this case we map the information that we want to transfer  Summary and Description, so everything that we want to have is being transferred, but as Remedy has additional fields/requirements that Jira does not cover  we cover them through ZigiOps. This way we provide all necessary data so that we can cover the data transfer successfully. If this is missing, Remedy denies the transfer and would not create anything. The transfer will fail.

Sometimes we use combinations of hard-coded values and values from the source system. For example, if we want to send the Jira task URL, as a comment in Remedy, but we do not have the full URL and we need to form it. It is a more complicated value  it is not a single URL, or a single value field that we can get from Jira. Instead, we combine a few values, plus a hard coded value.

So we get the Jira URL that we used to connect Jira to ZigiOps, then we manually add a hard coded string, and then we get the Key value from Jira. When you go to your Jira account, you choose a ticket and you can see that the URL of the ticket is the Jira URL, /browse/, and then the ticket key. This is why we do it this way in ZigiOps  we take the first part of the URL, the second one is hard coded, and the last one (the ticket key) comes dynamically from Jira. This allows us to get the right value for each ticket that we extract and send the correct link for each ticket in ServiceNow or Remedy, for example.

Simple field mapping

So far, we explained the simple mapping. Simple mapping means  1 filed on the left and 1 value on the right. The value can be an object from the target system, several objects one after another, hard-coded string, combination of objects, strings, etc. However, in the end we have 1 field and 1 value for it.

For most fields this works perfectly. However, there are fields that can have changing values (more than one value), depending on the situation.

These are for example the status fields (the status can be different  New, In progress, Resolved, etc.). One and the same field contains this value, and the value can change, so we have several options:

• We can set it so that it always has the same value (For example Status name in Jira = In Progress). This means that whatever happens, the status we send when we update Jira through the integration will always be In Progress. There are situations when this is valid.

• We can set an object, like {state} for example (from ServiceNow). The state field in ServiceNow can have different values, as well as the status name field in Jira. This is also simple mapping, as we have 1 field and 1 value for it. However, in this case we would have a problem because statuses that are equal from a user point of view, have different values in the two systems. In such cases the two systems cannot communicate, as the default values that they send are different. The value, for example that comes for state In Progess from ServiceNow is a number. In these cases we need to do conditional mapping.

• We can do conditional mapping  conditional mapping is enabled from the Condition option in ZigiOps. For each field in the mapping we are able to add conditions.

Conditional field mapping

With conditional mapping we can add conditions like: if something is true, change the value in this specific way and send it.

For example, we want to send status name to Jira. ServiceNow represents the status values as numbers, and the status In Progress in ServiceNow comes as 2. If we go to Jira, there the statuses are only To Do, In Progress and Done. These statuses do not match with what we have by default in ServiceNow.

One of the things we need to set conditions for is when the statuses differ  in ServiceNow it is New and in Jira it is To Do. There is no To Do in ServiceNow. The equivalent status is New in this case. So we need to tell ZigiOps  if the status in ServiceNow is New in Jira this status needs to be reported as To Do. When it is case sensitive, even if the status names match, again we will need to make a condition. However, if we save it in ServiceNow as In Progress and then we see the code of this particular ticket in the database, we have a state which is actually 2. It does not say In Progress. So when ZigiOps (or a client) collects this data (through the Rest API), it will see Status 2, and not In Progress.

All these things need to be planned at the time of forming the integration use case. If we, for example, leave it with simple mapping and match the status with the {state} in ServiceNow, it will send value 2. Jira will say: There is no such status and will ignore it.

What is a condition?

To avoid such problems, we define a condition instead.

We want to send to Jira status In Progress, but there is a condition. The condition is that the {state} value in ServiceNow needs to be 2. When we do this  if ZigiOps collects state value 2, we will change it to In Progress and then send it as a status name to Jira. If this is not true, this condition wont match, and we wont send anything. This value will be sent only if our condition is true.

But we can add more conditions. When {state} is 1, this equals to New in ServiceNow, that is the equivalent of To Do in Jira.

Then we can add another one  when {state} is 6 (in ServiceNow this equals to Resolved)  in Jira this is usually Done.

Now we already have conditional mapping.

Since the values that come from the source system do not match the correct values that the target system is expecting, we link the values and make these values readable between the systems.

This is not about formatting, or something related to the Transform phase, that is handled in the backend. There is no mismatch of the type, like one is XML, the other is JSON, and the third one is something else Here we have the values themselves that do not match, no matter of the code or type of communication that the systems use. The values will always be the same (it will be 2, no matter if it is using XML, JSON, etc.).

Advanced field mappings

We also have other situations, where we need more advanced mappings. For example, when we want to close an incident in ServiceNow.:

When we update the incident in ServiceNow, we do the same status conditional mapping, but in reverse  if the status in Jira is To Do we send 1, if the status is In Progress we send 2, and if the status is Done, we send 6 to ServiceNow.

There is something to keep in mind though. In Jira  when we click on Done it directly closes the task, but in ServiceNow, if we click on Resolve and try to save it, it says: Resolution code and Resolution note are mandatory. So we cannot resolve the incident if we do not provide these 2 things.

When it is Done in Jira and we want to resolve it in ServiceNow, as well, we can do the following mappings in ZigiOps:

• Send close_code to ServiceNow with value for example Closed/Resolved by Caller
• Add a closed_note, too, where the value can be anything, so we hard code it for example as Closed by Jira

However, if we do the above without a condition, these values will be sent to these fields in ServiceNow, no matter the status. Even if the status in Jira is not Done. But we want to send them only when the status in Jira is Done and this is why we can use conditional mapping. We will not change the values themselves, but we will add conditions when they will be sent and when not.

So we add the condition  if the status name in Jira is Done, only then send this value to this field. This would be the same for the closed_code and the closed_note fields that ServiceNow is requesting.

This way if the status in Jira is In Progress we will send the status, comments, attachments, etc, but we will not send these 2 fields.

If the status in Jira is set to Done we will send the status 6, all other changes, and also these 2 fields for closing.

The conditional mapping is not always with one condition. In ZigiOps we can have unlimited number of conditions.

This is how when one thing changes in Jira, this is a condition for 3 changes in ServiceNow.

Using such conditions, gives you huge freedom to match even super specific and complex integration use cases. With ZigiOps mapping you can literally do anything you need in your integration case.

Examples

Here are a few more specific examples:

• A case is created in ServiceNow, but in order for it to be updated, it needs to be assigned to a specific group. If it is not assigned to this specific group, we do not update it at all.
• Or if it is assigned to a specific group, we need to add an internal comment, and if it is assigned to a different group, we need to add a public comment.
• In Cherwell, instead of one field for Priority, there are 2 fields  Impact and Urgency. There is also Priority, that is defined automatically, based on Impact and Urgency. Jira, on the other hand, has only Priority.

Here is what we can do in cases like the last example with Cherwell. Based on these 2 values (Impact and Urgency) we can generate different priorities in another system, like Jira.

We can do 2 conditions: The Priority in Jira will be with value Highest, only when:

• Impact in Cherwell is Production AND
• Urgency in Cherwell is Critical

(Production + Critical = Highest)

But if the Impact is Production, and the Urgency is Medium then we will send High.

So the Impact is Production in both cases, but because of the second value, we change the value in Jira.

And another case: Impact is Test, and the Urgency is Critical, then we can send Low to Jira.

This is how we can take the two values  Impact and Urgency  and change the value in Jira, based on different combinations of them.

Jira does not have such fields, like Impact and Urgency, but the users in Cherwell, cannot directly change the Priority, they need to change the Impact and Urgency fields. This is why in ZigiOps we need to combine the 2 fields from Cherwell and send the correct values to Jira, through conditions.

Here is a little secret we will reveal now. ZigiOps clients will soon have the opportunity for even more advanced mapping. We are planning to add one more layer of mapping, so you will be able to have a whole mapping be executed in case a specific condition is true, and a whole different mapping executed, if it is not true. This is an upcoming feature that will be live in a few months.

What could go wrong  common field mapping mistakes

There are 3 most common mistakes that can happen during field mapping:

1. Missing value  there is a mandatory field and the client forgot to enter a value for it

1. Sending a non-existing field  the client defines a field that is wrong and does not exist in the target system.

1. Wrong value  the field exists, the client has defined it, but it is sending a wrong value.

Some of the fields in the systems have minimal requirements about what fields they need to receive values about for a given action (for example: in Jira to create a new Issue you necessarily need to send Summary).

ZigiOps provides a list of suggested values/fields, as well as some essential field validations. It detects some requirements of the target systems, and if they are not fulfilled it would highlight them.

Conclusion

Field mapping and doing your mapping right is critical for your integrations to work properly. You need to consider the different specifics of your integration use case and carefully plan and define all the mappings you need. Otherwise you might miss on some of the benefits of the integration, or worse  it might fail or work incorrectly.

This is why in ZigiOps we pay special attention to mapping, and we have pre-defined working mappings in our integration templates. If you want to see how it works live and ask us for the details of your specific use case, book a technical demo with our team. We will be happy to help with the planning of your integration flow and mapping.