Sync Engine synchronization mechanism basics

Advanced

Synchronization with external storages

The mechanism in Creatio for synchronization with external data storages is the Sync Engine. This mechanism enables you to create, modify, and delete Entity in the system based on synchronization with external systems and export data to external systems.

Synchronization is performed by using the SyncAgent class implemented in the Terrasoft.Sync namespace of the application kernel.

Classes used in the synchronization mechanism

Synchronization agent (SyncAgent) — a class with one public Synchronize method that triggers synchronization between storages.
Synchronization context (SyncContext) — a class representing the aggregation of providers and metadata for SyncAgent.
Storage — storage of synchronized data.
- Local storage (LocalProvider) — enables you to work with LocalItem in Creatio.
- External storage (RemoteProvider) — an external service or application from which data is synchronized with Creatio.
Synchronization item (SyncItem) — a set of objects from external and local storage which are synchronized.
- External storage synchronization item (RemoteItem) — represents a set of data from external storage that syncs automatically. It can consist of one or more entities (records) from the external storage.
- Synchronization item (SyncEntity) — a wrapper of a specific Entity. SyncEntity is required to work with Entity as the synchronizing object (add, delete, change).
- Synchronization item (LocalItem) — one or more objects from Creatio that are synchronized with the external storage as a unit. The synchronization item from the external storage, converted in the LocalItem entity contains a set of instances of the SyncEntity class.
SysSyncMetadata metadata table — contains service information of the synchronized elements and is essentially RemoteItem-LocalItem interchanges table.

General synchronization algorythm

Before starting synchronization, you must create an instance of SyncAgent and the SyncContext sync context, then update records in the metadata table with data from Creatio. For this, you need to call the CollectChangesInSyncedEntities class method that implements the IReplicaMetadata interface.

The algorithm for updating metadata records is the following:

If any previously synchronized entity in Creatio has been modified since the last synchronization, then the corresponding record in the metadata changes its modification date, the LocalState property is set to “Modified”, and the source of the modification is set to the LocalStore ID.
If a synchronized entity in Creatio has been deleted since the last synchronization — the corresponding record in the metadata LocalState is set to “Deleted".
If there is no corresponding record in the metadata for the entity in Creatio — it is ignored.

The process of synchronizing storages then starts the following:

All changes from the external storage are requested alternately.
You need to obtain the metadata for each item in the external storage .
1. If the metadata can not be obtained — this is a new item which should be converted to a Creatio element. To fill in a synchronization object, a FillLocalItem method is called from the specific RemoteItem instance. It is also recorded in the metadata (Id of the external storage, the element Id in the external storage, date of creation and modification is set as current, the source of creation and modification — external storage).
2. If the metadata is received, so this item has already been synchronized with Creatio. You need to go to the version conflict resolution. By default, the last change in the application or external storage (RemoteProvider) has the priority.
3. The metadata for the current pair of synchronization items is actualized.

After looking through all the modified items from the external storage, the elements that were changed in Creatio, but was not changed in the external storage remain in the metadata (in the interval between the last and the current synchronization).

You need to get elements changed in Creatio in the interval between the last synchronization and the current synchronization.
Save the changes in the external storage.
You must update the modification date of the items in the metadata (Creatio is the change source).

After that, you need to add new, not synchronized records to the external storage, and add metadata for new items.

Synchronization description

An activity and participants are synchronized into one Google-calendar task. An activity (Activity) and participants (SyncEntity) are one element of the synchronization - LocalItem.

RemoteItem - Google task received outside Creatio. LocalItem - a set of objects (SyncEntity), to which the Google task is converted.

The synchronization schema:

Working with synchronization metadata

The auxiliary SysSyncMetaData metadata table is used for synchronization, which is the junction between the outer RemoteItem table (synchronizing element in external storage) and LocalItem (synchronization element in Creatio). Each table row is represented in the system as an instance of SysSyncMetaData class.

Only information about synchronized elements is stored in the metadata.

There can be multiple metadata table records for a single synchronization element - one for each application object included in a synchronization element.

Activity and participants — a single synchronization element. However, the metadata contains one record for each activity and one record for each participant.

Currently, only one object from external storage is transformed into several Creatio objects:

scr_syncengine_metadatadescription_transformation_scheme.png

The metadata system for a single synchronization element is represented as the ItemMetadata object class (SysSyncMetaData element collection). Metadata management is carried out through the class that implements the IReplicaMetadata interface. An instance of the class that implements the IReplicaMetadata interface is created via the MetaDataStore factory class for a particular storage.

SyncContext class

Advanced

Namespace Terrasoft.Sync

A class representing the aggregation of providers and metadata for SyncAgent.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the SyncContext class.

Constructors

SyncContext()

Creates a class instance.

Properties

CurrentSyncStartVersion DateTime

The current date and time synchronization in UTC. Set after the metadata update.

LastSyncVersion DateTime

The date and time of the last synchronization in UTC.

LocalProvider LocalProvider

Enables you to work with LocalItem.

Logger ISyncLogger

The object that enables messages to be saved into the integration log.

MsgLogger Terrasoft.Sync.SyncMsgLogger

Message logger for synchronization engine.

RemoteProvider RemoteProvider

External service or application, data from which is synchronized with Creatio.

ReplicaMetadata IReplicaMetadata

Works with metadata.

UserConnection UserConnection

User connection.

Methods

void LogError(SyncAction operation, SyncDirection direction, System.string format, params System.object[] args)
void LogError(SyncAction operation, SyncDirection direction, System.string format, System.Exception exception, params System.object[] args)

Writes error message to the log.

Parameters

operation	Log action for an object synchronization.
direction	Synchronization direction.
format	Format.
exception	Exception that caused this error.
args	Format parameters.

void LogInfo(SyncAction operation, SyncDirection direction, System.string format, params System.object[] args)

Writes information message to the log.

Parameters

operation	Log action for an object synchronization.
direction	Synchronization direction.
format	Format.
args	Format parameters.

void SetLookupColumnValue(Entity entity, System.string lookupColumnName, System.string lookupDisplayValue)
void SetLookupColumnValue(Entity entity, System.string lookupColumnName, System.string lookupDisplayValue, System.string lookupDisplayColumnName)

Sets entity lookup column value.

Parameters

entity	Instance of the Terrasoft.Core.Entities.Entity class.
lookupColumnName	Lookup column name.
lookupDisplayValue	Lookup column display value.
lookupDisplayColumnName	Lookup display column name.

RemoteProvider class

Advanced

Namespace Terrasoft.Sync

RemoteProvider — a basic class that enables you to work with an external storage. In fact, it is the only way to work with external systems.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the RemoteProvider class.

Properties

StoreId Guid

ID of external storage that will be synchronized.

Version DateTime

Date and time of the last synchronization in UTC.

SyncItemSchemaCollection List<SyncItemSchema>

External storage element schema.

RemoteChangesCount Int

Number of items processed from external storage.

LocalChangesCount Int

Number of items processed from local storage.

Methods

abstract IEnumerable<Type> KnownTypes()

Returns a collection of all types that implement the IRemoteItem interface. SyncAgent builds the SyncItemSchema instances that describe the entities that participate in synchronization.

abstract void ApplyChanges(SyncContext context, IRemoteItem synItem)

Applies changes to external storage element.

Parameters

context	Synchronization context.
syncItem	Item of the synchronization.

abstract void CommitChanges(SyncContext context)

Called after processing changes in external and local storage. Intended for the implementation of necessary additional steps for the specific integration implementation.

abstract IEnumerable<IRemoteItem> EnumerateChanges(SyncContext context)

Returns an enumeration of new and modified elements of external storage.

abstract IRemoteItem LoadSyncItem(SyncItemSchema schema, string id)
virtual IRemoteItem LoadSyncItem(SyncItemSchema schema, ItemMetadata itemMetadata)

Fills in the IRemoteItem instance with data from external storage.

Parameters

schema	Synchronization schema.
id	Unique foreign key.
itemMetadata	Synchronization metadata.

abstract IRemoteItem CreateNewSyncItem(SyncItemSchema schema)

Creates a new instance of IRemoteItem.

abstract IEnumerable<LocalItem> CollectNewItems(SyncContext context)

Returns an enumeration of new Creatio entities that will be synchronized with external storage.

virtual SyncConflictResolution ResolveConflict(IRemoveItem syncItem, ItemMetadata itemMetaData, Guid localStoreId)

Resolves conflicts between changed elements of local and external storages. By default, (RemoteProvider) priority is given to changes in Creatio.

virtual bool NeedMetaDataActualization()

Returns the sign that checks whether there is a need to update the metadata before starting synchronization.

virtual IEnumerable<ItemMetadata> GetLocallyModifiedItemsMetadata(SyncContext context, EntitySchemaQuery modifiedItemsEsq)

Returns synchronization elements changed in the local store since the last synchronization.

Parameters

context	Synchronization Context.
modifiedItemsEsq	Request to the scheme object Terrasoft.Core.Configuration.SysSyncMetaData.

IRemoteItem interface

Advanced

Namespace Terrasoft.Sync

Interface of the external storage element.

The class that implements the IRemoteItem interface is an indivisible unit of synchronization and represents one element of the synchronization of the external data storage. This class is a container for data coming from an external system, and it knows how to convert the data to the Entity entity, and vice versa. The interface contains two methods: FillLocalItem and FillRemoteItem for converting external synchronization elements (IRemoteItem) to LocalItem, and vice versa.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the members of the IRemoteItem interface.

Methods

void FillLocalItem(SyncContext context, ref LocalItem localItem)

Fills in properties of an element of the LocalItem local storage with values of external storage element. Used to apply changes in local storage.

void FillRemoteItem(SyncContext context, ref LocalItem localItem)

Fills in properties of an element of external storage with element values from the LocalItem local storage. Used to apply changes in external storage.

MapAttribute class

Namespace Terrasoft.Sync

The Map attribute decorates the IRemoteItem interface implementations.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the MapAttribute class.

Properties

SchemaName string

Object schema name.

Order int

The entity processing order for the synchronization element.

IsPrimarySchema bool

A flag that indicates that this schema is a key element of this synchronization element. It can be installed in one schema only.

PrimarySchemaName string

The schema name of the main object. It can not be set in tandem with IsPrimarySchema.

ForeignKeyColumnName string

The column name for the connection with the details of the main object It can not be set in tandem with IsPrimarySchema.

Direction SyncDirection

It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload.

If it contains the Download flag - the changes will not apply to Creatio.

If it does not contain the Upload flag - the new entities will not be selected from Creatio.

FetchColumnNames String[]

The names of the columns that will be loaded from the local storage.

Examples of using map attribute

SchemaName is the main parameter. This is the name of the EntitySchema that is included in the current synchronization element.

[Map("Activity", 0)]
[Map("ActivityParticipant", 1)]
public class GoogleTask: IRemoteItem {
    ...

This class declaration task from Google Calendar will sync with the activity and its participants from Creatio.

The second parameter, Order, specifies in which order Entity will be stored in the local storage. Activityis indicated first, because ActivityParticipant stores a link to the created activity.

In most cases, SyncAgent can automatically generate a request for a sample of the new elements of synchronization with Creatio. To do this, you must specify the basic entity and method of communication with the details:

[Map("Activity", 0, IsPrimarySchema = true)]
[Map("ActivityParticipant", 1, PrimarySchemaName = "Activity", ForeingKeyColumnName = "Activity")]
public class GoogleTask: IRemoteItem {
    ...

In this case, a request for new activities will be sent to the database along with a request for each selected activity to receive their participants.

LocalProvider class

Advanced

Namespace Terrasoft.Sync.

Local Storage (LocalProvider) - encapsulates the work with data in internal storage (Creatio).

LocalProvider - basic class that implements work with the local storage. Enables you to work with LocalItem.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the LocalProvider class.

Constructors

public LocalProvider(UserConnection userConnection)

Creates a class instance using UserConnection.

Properties

StoreId Guid

ID of Local storage that will be synchronized.

MaxItemsPerSelect int

Maximum items in select count.

UserConnection UserConnection

UserConnection instance.

Methods

static void AddItemSchemaColumns(EntitySchemaQuery esqForFetching, EntityConfig entityConfig)

Adds EntitySchemaQuery column specified in EntityConfig.

Parameters

esqForFetching	Instance of the Terrasoft.Core.Entities.EntitySchemaQuery class.
entityConfig	Instance of the Terrasoft.Sync.EntityConfig class.

void ApplyChanges(SyncContext context, LocalItem entities)

Applies changes to each SyncEntity in LocalItem.

Parameters

context	Synchronization context.
entities	Items of the synchronization.

LocalItem FetchItem(ItemMetadata itemMetaData, SyncItemSchema itemSchema, bool loadAllColumns = false)

Loads a collection of entities associated with a particular synchronization element.

Parameters

itemMetaData	Synchronization element.
itemSchema	Synchronization schema.
loadAllColumns	If true, loads all entities columns, loads only columns from entity EntityConfig instance.

SyncEntity class

Advanced

Namespace Terrasoft.Sync.

The SyncEntity class encapsulates Entity instance and all the necessary actions to perform the synchronization of the instance properties.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the SyncEntity class.

Properties

EntitySchemaName string

The name of the schema for which the wrapper is created.

Entity Terrasoft.Core.Entities.Entity

Entity for which the wrapper is created.

State Terrasoft.Sync.SyncState

The last action performed on the entity (0 - not changed, 1 - new, 2 - changed 3 - deleted).

Action Terrasoft.Sync.SyncAction

Action to be done with the entity.

ExtraParameters string

Additional information that can be bound to a specific entity by an external provider for a specific purpose.

SyncItemSchema class

Advanced

Namespace Terrasoft.Sync

Synchronization element entity settings schema.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the SyncItemSchema class.

Properties

SyncValueName string

Entity type name.<

SyncValueType Type

Entity type.

PrimaryEntityConfig Terrasoft.Sync.EntityConfig

Synchronization element entity settings.

Configs List<EntityConfig>

Synchronization element entity settings list.

DetailConfigs List<DetailEntityConfig>

Synchronization element detail entity settings list.

Direction Terrasoft.Sync.SyncDirection

It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload.

If it does not contain the Download flag, changes will not be applied in Creatio.

If it does not contain the Upload flag, new entities will not be selected from Creatio.

Methods

static SyncItemSchema CreateSyncItemSchema(Type syncValueType)

It creates a configuration element synchronization entity with all the settings of the related entities.

void Validate(UserConnection userConnection)

The method checks that EntityConfig is well-formed.

If authentication fails, an exception is applied. If the EntityConfig schema name is specified twice, DublicateDataException is generated. If the name of the defunct schema is specified, InvalidSyncItemSchemaException is generated).

EntityConfig class

Advanced

Namespace Terrasoft.Sync.

Synchronization element entity settings.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the EntityConfig class.

Properties

SchemaName string

Object schema name.

Order int

The order of processing entities for a synchronization element. The lower the value, the sooner the entity will be processed in the processing of the synchronization element.

Direction Terrasoft.Sync.SyncDirection

It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload.

If it does not contain the Download flag, changes will not be applied in Creatio.

If it does not contain the Upload flag, new entities will not be selected from Creatio.

FetchColumnNames string[]

The names of the columns that will be loaded from the local storage. If the value is not specified, it will load all object columns.

DetailEntityConfig class

Advanced

Namespace Terrasoft.Sync.

Synchronization element detail entities settings.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the DetailEntityConfig class.

Properties

PrimarySchemaName string

Creatio main synchronized entity schema name.

ForeignKeyColumnName string

The column name for the connection with the details of the main object.

LocalItem class

Advanced

Namespace Terrasoft.Sync.

One or more objects from Creatio that are synchronized with external storage as a unit. It contains a set of SyncEntity class instances.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the LocalItem class.

Properties

Entities Dictionary<string, List<SyncEntity>>

The SyncEntity collection that is set in accordance with a SyncItem. It contains a collection of "key-value" pairs, where the key is the schema name, and the value is the SyncEntity collection of the scheme.

Version DateTime

The highest value of the date and time of the modification of all Entities in LocalItem.

Schema SyncItemSchema

Synchronization element entity settings schema.

Methods

void AddOrReplace(string schemaName, SyncEntity syncEntity)

Adds new SyncEntity to the collection. If SyncEntity with EntityId already exists, it replaces it.

SyncEntity Add(UserConnection userConnection, string schemaName)

Creates and adds a new SyncEntity collection.

SysSyncMetaData class

Advanced

Properties

RemoteId string

Element ID in external storage.

SyncSchemaName string

Synchronized element schema name.

LocalId Guid

Element ID in local storage.

IsLocalDeleted bool

It indicates whether an item has been removed from the local storage since the last synchronization. The parameter is updated before the synchronization and application of changes in the local storage. On the basis of its value, when selecting modified elements from local storage, the SyncEntity state is set. Obsolete, left for compatibility. LocalState is currently used.

IsRemoteDeleted bool

It indicates whether an item has been removed from the external storage since the last synchronization. Obsolete, left for compatibility. RemoteState is currently used.

Version Date

Date of the last element modification.

ModifiedInStoreId Guid

Id of storage in which the last modification was performed.

CreatedInStoreId Guid

Id of storage in which the synchronization element was created.

RemoteStoreId Guid

Id of external storage with which the element was synchronized.

ExtraParameters string

Additional element parameters.

LocalState int

Element state in local storage (0 - not modified, 1 - new, 2 - modified, 3 - deleted).

RemoteState int

Element state in external storage (0 - not modified, 1 - new, 2 - modified, 3 - deleted).

MetaDataStore class

Advanced

Namespace Terrasoft.Sync.

Creates the specific class instance that implements the IReplicaMetadata interface for a storage.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the MetaDataStore class.

Methods

IReplicaMetadata GetReplicaMetadata(Guid localStoreId, Guid remoteStoreId)

Creates the class instance that implements the IReplicaMetadata interface for the specific storage.

Parameters

localStoreId	Local storage id.
remoteStoreId	Remote storage id.

void RemoveReplicaMetadata(Guid storageId)

Removes all metadata for storage.

ItemMetaData class

Advanced

Namespace Terrasoft.Sync.

This class is an indivisible object of metadata synchronization. It contains a set of metadata for each synchronization element (SysSyncMetadata element collection).

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the methods, properties, parent classes, and implemented interfaces of the ItemMetaData class.

Properties

RemoteId string

Element ID in external storage.

RemoteItemName string

Element name in external storage.

IReplicaMetadata interface

Advanced

Namespace Terrasoft.Sync.

Class implementing the IReplicaMetadata interface, encapsulates the synchronization metadata and works with ItemMetadata objects.

Note. Use the “.NET class libraries of platform core” documentation to access the full list of the members of the IReplicaMetadata interface.

Properties

RemoteStoreId Guid

External storage ID.

LocalStoreId Guid

Local storage ID.

Methods

ItemMetadata FindItemMetaData(string remoteItemId)

Finds and returns the ItemMetadata synchronization metadata object by an ID in the remoteItemId external storage.

void UpdateItemMetadata(ItemMetadata oldItemMetaDatas, IRemoteItem remoteItem, LocalItem localItem, bool changesToBpm)

Updates metadata after synchronization.

Parameters

oldItemMetaDatas	Metadata instance from previous synchronization.
remoteItem	Remote item instance.
localItem	Local item instance.
changesToBpm	Current synchronization direction.

IEnumerable<ItemMetadata> EnumerateItemsWithChangesInBpm (SyncContext context)

Returns a collection of ItemMetadata objects that have been modified since the last synchronization and not processed during the current synchronization session.

void CollectChangesInSyncedEntities (UserConnection userConnection, string schemaName, DateTime lastSyncVersion)

Updates metadata for synchronization elements modified in Creatio. If an element has been modified since the last SysMetadata synchronization, the Version column will be filled in with the date of element modification. The ActualizeSysSyncMetaData procedure is used to update metadata.

Parameters

userConnection	Terrasoft.Core.UserConnection Instance.
schemaName	Sync entity name.
lastSyncVersion	Last synchronization date.

void CollectNewDetailsForSyncedEntities (UserConnection userConnection, DetailEntityConfig detailEntityConfig, string remoteItemName, DateTime lastSyncVersion)

Creates new records in the SysSyncMetaData table for the synchronization element details.

Parameters

userConnection	User connection.
detailEntityConfig	Instance of the Terrasoft.Sync.DetailEntityConfig class.
remoteItemName	Remote storage item name.
lastSyncVersion	Last synchronization date.

bool TryResolveRemoteId (Guid localId, out string remoteId)

Returns the element ID in the external remoteId storage from metadata by a unique localId synchronization element in Creatio. If an element is marked as deleted, the remoteId wil not be returned, and the method will return false.

bool TryResolveExtraParameters (Guid localId, out string extraParameters) Boolean

Returns additional extraParameters parameters for the synchronization element by localId. If extraParameters are not found, the method returns false.