Creatio synchronization with external storages

General information

The mechanism in Creatio for synchronization with external data storages is the Sync Engine. This mechanism enables you to create, modify, and delete Entity in the system based on synchronization with external systems and export data to external systems.

Synchronization is performed by using the SyncAgent class implemented in the Terrasoft.Sync namespace of the application kernel.

Classes used in the synchronization mechanism

• Synchronization agent (SyncAgent) — a class with one public Synchronize method that triggers synchronization between storages.
• Synchronization context (SyncContext) — a class representing the aggregation of providers and metadata for SyncAgent.
• Storage — storage of synchronized data.
  • Local storage (LocalProvider) — enables you to work with LocalItem in Creatio.
  • External storage (RemoteProvider) — an external service or application from which data is synchronized with Creatio.
• Synchronization item (SyncItem) — a set of objects from external and local storage which are synchronized.
  • External storage synchronization item (RemoteItem) — represents a set of data from external storage that syncs automatically. It can consist of one or more entities (records) from the external storage.
  • Synchronization item (SyncEntity) — a wrapper of a specific Entity. SyncEntity is required to work with Entity as the synchronizing object (add, delete, change).
  • Synchronization item (LocalItem) — one or more objects from Creatio that are synchronized with the external storage as a unit. The synchronization item from the external storage, converted in the LocalItem entity contains a set of instances of the SyncEntity class.
• SysSyncMetadata metadata table — contains service information of the synchronized elements and is essentially RemoteItem-LocalItem interchanges table. Metadata sync description can be found in the "Synchronizing metadata in Creatio" article.

General synchronization algorythm

Before starting synchronization, you must create an instance of SyncAgent and the SyncContext sync context, then update records in the metadata table with data from Creatio. For this, you need to call the CollectChangesInSyncedEntities class method that implements the IReplicaMetadata interface.

The algorithm for updating metadata records is the following:

1. If any previously synchronized entity in Creatio has been modified since the last synchronization, then the corresponding record in the metadata changes its modification date, the LocalState property is set to “Modified”, and the source of the modification is set to the LocalStore ID.
2. If a synchronized entity in Creatio has been deleted since the last synchronization — the corresponding record in the metadata LocalState is set to “Deleted".
3. If there is no corresponding record in the metadata for the entity in Creatio — it is ignored.

The process of synchronizing storages then starts the following:

1. All changes from the external storage are requested alternately.
2. You need to obtain the metadata for each item in the external storage .
   1. If the metadata can not be obtained — this is a new item which should be converted to a Creatio element. To fill in a synchronization object, a FillLocalItem method is called from the specific RemoteItem instance. It is also recorded in the metadata (ID of the external storage, the element ID in the external storage, date of creation and modification is set as current, the source of creation and modification — external storage).
   2. If the metadata is received, so this item has already been synchronized with Creatio. You need to go to the version conflict resolution. By default, the last change in the application or external storage (RemoteProvider) has the priority.
   3. The metadata for the current pair of synchronization items is actualized.

After looking through all the modified items from the external storage, the elements that were changed in Creatio, but was not changed in the external storage remain in the metadata (in the interval between the last and the current synchronization).

1. You need to get elements changed in Creatio in the interval between the last synchronization and the current synchronization.
2. Save the changes in the external storage.
3. You must update the modification date of the items in the metadata (Creatio is the change source).

After that, you need to add new, not synchronized records to the external storage, and add metadata for new items.

The synchronization context

SyncContext class

A class representing the aggregation of providers and metadata for SyncAgent. The properties of the SyncContext class are presented in Table 1.

Table 1. SyncContext class properties

Requirements for synchronization with external storage

External storage

External storage (RemoteProvider) — encapsulates data from the external storage.

RemoteProvider — a basic class that enables you to work with an external storage. In fact, it is the only way to work with external systems. Properties of the RemoteProvider class are presented in table 2 and the methods — in table 3.

Table 2. RemoteProvider class properties

Table 3. RemoteProvider class methods

IRemoteItem interface

The class that implements the IRemoteItem interface is an indivisible unit of synchronization and represents one element of the synchronization of the external data storage. This class is a container for data coming from an external system, and it knows how to convert the data to the Entity entity, and vice versa. The interface contains two methods: FillLocalItem and FillRemoteItem for converting external synchronization elements (RemoteItem) to LocalItem, and vice versa. Interface methods are presented in Table 4.

Table 4. IRemoteItem interface methods

Map attribute

The Map attribute decorates the iRemoteItem interface implementations. SchemaName is the main parameter. This is the name of the EntitySchema that is included in the current synchronization element.

[Map("Activity", 0)]
[Map("ActivityParticipant", 1)]
public class GoogleTask: IRemoteItem {
. . .

This class declaration task from Google Calendar will sync with the activity and its participants from Creatio.

The second parameter, Order, specifies in which order Entity will be stored in the local storage. Activity is indicated first, because ActivityParticipant stores a link to the created activity.

In most cases, SyncAgent can automatically generate a request for a sample of the new elements of synchronization with Creatio. To do this, you must specify the basic entity and method of communication with the details:

[Map("Activity", 0, IsPrimarySchema = true)]
[Map("ActivityParticipant", 1, PrimarySchemaName = "Activity", ForeingKeyColumnName = "Activity")]
public class GoogleTask: IRemoteItem {
. . .

In this case, a request for new activities will be sent to the database along with a request for each selected activity to receive their participants. The attribute properties list is presented in Table 5.

Table 5. Map attribute properties

Local storage

Local Storage (LocalProvider) - encapsulates the work with data in internal storage (Creatio).

LocalProvider - basic class that implements work with the local storage. Enables you to work with LocalItem. Methods of this class are immutable and are listed in Table 6.

Table 6. LocalProvider class methods

SyncEntity class

The class encapsulates SyncEntity Entity instance and all the necessary actions to perform the synchronization of the instance properties. Class Properties are summarized in Table 7.

Table 7. SyncEntity class properties

SystemSchema class

Synchronization element entity settings schema. Class properties are shown in Table 8 and methods in Table 9.

Table 8. SyncItemSchema class properties

Table 9. SyncItemSchema class methods

EntityConfig Class

Synchronization element entity settings. Class Properties are summarized in Table 10.

Table 10. EntityConfig class properties

DetailEntityConfig class

Synchronization element detail entities settings. Class properties are displayed on Table 11.

Table 11. DetailEntityConfig class properties

LocalItem class

One or more objects from Creatio that are synchronized with external storage as a unit. It contains a set of SyncEntity class instances. Class properties are shown in Table 12 and methods in Table 13.

Table 12. LocalItem class properties

Table 13. LocalItem class methods

Synchronization example

An activity and participants are synchronized into one Google-calendar task. An activity (Activity) and participants (SyncEntity) are one element of the synchronization - LocalItem.

RemoteItem - Google task received outside Creatio. LocalItem - a set of objects (SyncEntity), to which the Google task is converted.

The synchronization schema is displayed on Fig. 1.

Fig. 1. Synchronization schema