Bpm'online synchronization with external storages
Glossary Item Box
General information
The mechanism in bpm'online for synchronization with external data storages is the Sync Engine. This mechanism enables you to create, modify, and delete Entity in the system based on synchronization with external systems and export data to external systems.
Synchronization is performed by using the SyncAgent class implemented in the Terrasoft.Sync namespace of the application kernel.
Classes used in the synchronization mechanism
- Synchronization agent (SyncAgent) — a class with one public Synchronize method that triggers synchronization between storages.
- Synchronization context (SyncContext) — a class representing the aggregation of providers and metadata for SyncAgent.
- Storage — storage of synchronized data.
- Local storage (LocalProvider) — enables you to work with LocalItem in bpm'online.
- External storage (RemoteProvider) — an external service or application from which data is synchronized with bpm'online.
- Synchronization item (SyncItem) — a set of objects from external and local storage which are synchronized.
- External storage synchronization item (RemoteItem) — represents a set of data from external storage that syncs automatically. It can consist of one or more entities (records) from the external storage.
- Synchronization item (SyncEntity) — a wrapper of a specific Entity. SyncEntity is required to work with Entity as the synchronizing object (add, delete, change).
- Synchronization item (LocalItem) — one or more objects from bpm'online that are synchronized with the external storage as a unit. The synchronization item from the external storage, converted in the LocalItem entity contains a set of instances of the SyncEntity class.
- SysSyncMetadata metadata table — contains service information of the synchronized elements and is essentially RemoteItem-LocalItem interchanges table. Metadata sync description can be found in the "Synchronizing metadata in bpm'online" article.
General synchronization algorythm
Before starting synchronization, you must create an instance of SyncAgent and the SyncContext sync context, then update records in the metadata table with data from bpm'online. For this, you need to call the CollectChangesInSyncedEntities class method that implements the IReplicaMetadata interface.
The algorithm for updating metadata records is the following:
- If any previously synchronized entity in bpm'online has been modified since the last synchronization, then the corresponding record in the metadata changes its modification date, the LocalState property is set to “Modified”, and the source of the modification is set to the LocalStore ID.
- If a synchronized entity in bpm'online has been deleted since the last synchronization — the corresponding record in the metadata LocalState is set to “Deleted".
- If there is no corresponding record in the metadata for the entity in bpm'online — it is ignored.
The process of synchronizing storages then starts the following:
- All changes from the external storage are requested alternately.
- You need to obtain the metadata for each item in the external storage .
- If the metadata can not be obtained — this is a new item which should be converted to a bpm'online element. To fill in a synchronization object, a FillLocalItem method is called from the specific RemoteItem instance. It is also recorded in the metadata (ID of the external storage, the element ID in the external storage, date of creation and modification is set as current, the source of creation and modification — external storage).
- If the metadata is received, so this item has already been synchronized with bpm'online. You need to go to the version conflict resolution. By default, the last change in the application or external storage (RemoteProvider) has the priority.
- The metadata for the current pair of synchronization items is actualized.
After looking through all the modified items from the external storage, the elements that were changed in bpm'online, but was not changed in the external storage remain in the metadata (in the interval between the last and the current synchronization).
- You need to get elements changed in bpm'online in the interval between the last synchronization and the current synchronization.
- Save the changes in the external storage.
- You must update the modification date of the items in the metadata (bpm'online is the change source).
After that, you need to add new, not synchronized records to the external storage, and add metadata for new items.
The synchronization context
SyncContext class
A class representing the aggregation of providers and metadata for SyncAgent. The properties of the SyncContext class are presented in Table 1.
Table 1. SyncContext class properties
| Field | Type | Purpose |
|---|---|---|
| Logger | ISyncLogger | The object that enables messages to be saved into the integration log. |
| LocalProvider | LocalProvider | Enables you to work with LocalItem. |
| RemoteProvider | RemoteProvider | External service or application, data from which is synchronized with bpm'online. |
| ReplicaMetadata | IReplicaMetadata | Works with metadata. |
| LastSyncVersion | DateTime | The date and time of the last synchronization in UTC. |
| CurrentSyncStartVersion | DateTime | The current date and time synchronization in UTC. Set after the metadata update. |
Requirements for synchronization with external storage
External storage
External storage (RemoteProvider) — encapsulates data from the external storage.
RemoteProvider — a basic class that enables you to work with an external storage. In fact, it is the only way to work with external systems. Properties of the RemoteProvider class are presented in table 2 and the methods — in table 3.
Table 2. RemoteProvider class properties
| Field | Type | Description |
|---|---|---|
| StoreId | Guid | ID of external storage that will be synchronized. |
| Version | DateTime | Date and time of the last synchronization in UTC. |
| SyncItemSchemaCollection | List | External storage element schema |
| RemoteChangesCount | Int | Number of items processed from external storage. |
| LocalChangesCount | Int | Number of items processed from local storage. |
Table 3. RemoteProvider class methods
| Method | Returning value type | Description |
|---|---|---|
| KnownTypes() | IEnumerable | Returns a collection of all types that implement the IRemoteItem interface. SyncAgent builds the SyncItemSchema instances that describe the entities that participate in synchronization. |
| ApplyChanges(SyncContext context, IRemoteItem synItem) | Void | Applies changes to external storage element. |
| CommitChanges(SyncContext context) | Void | Called after processing changes in external and local storage. Intended for the implementation of necessary additional steps for the specific integration implementation. |
| EnumerateChanges(SyncContext context) | IEnumerable | Returns an enumeration of new and modified elements of external storage. |
| LoadSyncItem(SyncItemSchema schema, string id) | IRemoteItem | Fills in the IRemoteItem instance with data from external storage. |
| CreateNewSyncItem(SyncItemSchema schema) | IRemoteItem | Creates a new instance of IRemoteItem. |
| CollectNewItems(SyncContext context) | IEnumerable | Returns an enumeration of new bpm'online entities that will be synchronized with external storage. |
| ResolveConflict(IRemoveItem syncItem, ItemMetadata itemMetaData, Guid localStoreId) | SyncConflictResolution | Resolves conflicts between changed elements of local and external storages. By default, (RemoteProvider) priority is given to changes in bpm'online. |
| NeedMetaDataActualization() | Boolean | Returns the sign that checks whether there is a need to update the metadata before starting synchronization. |
| GetLocallyModifiedItemsMetadata(SyncContext context, EntitySchemaQuery modifiedItemsEsq) | IEnumerable | Returns synchronization elements changed in the local store since the last synchronization. |
IRemoteItem interface
The class that implements the IRemoteItem interface is an indivisible unit of synchronization and represents one element of the synchronization of the external data storage. This class is a container for data coming from an external system, and it knows how to convert the data to the Entity entity, and vice versa. The interface contains two methods: FillLocalItem and FillRemoteItem for converting external synchronization elements (RemoteItem) to LocalItem, and vice versa. Interface methods are presented in Table 4.
Table 4. IRemoteItem interface methods
| Method | Returning value type | Description |
|---|---|---|
| FillLocalItem(SyncContext context, ref LocalItem localItem) | Void | Fills in properties of an element of the LocalItem local storage with values of external storage element. Used to apply changes in local storage. |
| FillRemoteItem(SyncContext context, ref LocalItem localItem) | Void | Fills in properties of an element of external storage with element values from the LocalItem local storage. Used to apply changes in external storage. |
Map attribute
The Map attribute decorates the iRemoteItem interface implementations. SchemaName is the main parameter. This is the name of the EntitySchema that is included in the current synchronization element.
[Map("Activity", 0)] [Map("ActivityParticipant", 1)] public class GoogleTask: IRemoteItem { . . .
This class declaration task from Google Calendar will sync with the activity and its participants from bpm'online.
The second parameter, Order, specifies in which order Entity will be stored in the local storage. Activity is indicated first, because ActivityParticipant stores a link to the created activity.
In most cases, SyncAgent can automatically generate a request for a sample of the new elements of synchronization with bpm'online. To do this, you must specify the basic entity and method of communication with the details:
[Map("Activity", 0, IsPrimarySchema = true)] [Map("ActivityParticipant", 1, PrimarySchemaName = "Activity", ForeingKeyColumnName = "Activity")] public class GoogleTask: IRemoteItem { . . .
In this case, a request for new activities will be sent to the database along with a request for each selected activity to receive their participants. The attribute properties list is presented in Table 5.
Table 5. Map attribute properties
| Parameter | Type | Description |
|---|---|---|
| SchemaName | String | Object schema name. |
| Order | Int | The entity processing order for the synchronization element. |
| IsPrimarySchema | Boolean | A flag that indicates that this schema is a key element of this synchronization element. It can be installed in one schema only. |
| PrimarySchemaName | String | The schema name of the main object. It can not be set in tandem with IsPrimarySchema. |
| ForeignKeyColumnName | String |
The column name for the connection with the details of the main object It can not be set in tandem with IsPrimarySchema. |
| Direction | SyncDirection |
It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload. If it contains the Download flag - the changes will not apply to bpm'online. If it does not contain the Upload flag - the new entities will not be selected from bpm'online. |
| FetchColumnNames | String[] | The names of the columns that will be loaded from the local storage. |
Local storage
Local Storage (LocalProvider) - encapsulates the work with data in internal storage (bpm'online).
LocalProvider - basic class that implements work with the local storage. Enables you to work with LocalItem. Methods of this class are immutable and are listed in Table 6.
Table 6. LocalProvider class methods
| Method | Returning value type | Description |
|---|---|---|
| AddItemSchemaColumns(EntitySchemaQuery esqForFetching, EntityConfig entityConfig) | Void | Adds EntitySchemaQuery column specified in EntityConfig. |
| ApplyChanges(SyncContext context, LocalItem entities) | Void | Applies changes to each SyncEntity in LocalItem. |
| FetchItem(ItemMetadata itemMetaData, SyncItemSchema itemSchema, bool loadAllColumns = false) | LocalItem | Loads a collection of entities associated with a particular synchronization element. |
SyncEntity class
The class encapsulates SyncEntity Entity instance and all the necessary actions to perform the synchronization of the instance properties. Class Properties are summarized in Table 7.
Table 7. SyncEntity class properties
| Parameter | Type | Description |
|---|---|---|
| EntitySchemaName | String | The name of the schema for which the wrapper is created. |
| Entity | Entity | Entity for which the wrapper is created. |
| State | SyncState | The last action performed on the entity (0 - not changed, 1 - new, 2 - changed 3 - deleted). |
SystemSchema class
Synchronization element entity settings schema. Class properties are shown in Table 8 and methods in Table 9.
Table 8. SyncItemSchema class properties
| Parameter | Type | Description |
|---|---|---|
| SyncValueName | String | Entity type name |
| SyncValueType | Type | Entity type |
| PrimaryEntityConfig | EntityConfig | Synchronization element entity settings. |
| Configs | List | Synchronization element entity settings list. |
| DetailConfigs | List | Synchronization element detail entity settings list. |
| Direction | SyncDirection |
It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload. If it does not contain the Download flag, changes will not be applied in bpm'online. If it does not contain the Upload flag, new entities will not be selected from bpm'online. |
Table 9. SyncItemSchema class methods
| Method | Returning value type | Description |
|---|---|---|
| CreateSyncItemSchema(Type syncValueType) | SyncItemSchema | It creates a configuration element synchronization entity with all the settings of the related entities. |
| Validate(UserConnection userConnection) | Void |
The method checks that EntityConfig is well-formed. If authentication fails, an exception is applied. If the EntityConfig schema name is specified twice, DublicateDataException is generated. If the name of the defunct schema is specified, InvalidSyncItemSchemaException is generated). |
| FetchItem(ItemMetadata itemMetaData, SyncItemSchema itemSchema, bool loadAllColumns = false) | LocalItem | Loads a collection of entities associated with a particular synchronization element. |
EntityConfig Class
Synchronization element entity settings. Class Properties are summarized in Table 10.
Table 10. EntityConfig class properties
| Parameter | Type | Description |
|---|---|---|
| SchemaName | String | Object schema name. |
| Order | Int | The order of processing entities for a synchronization element. The lower the value, the sooner the entity will be processed in the processing of the synchronization element. |
| Direction | SyncDirection |
It specifies the synchronization direction for the objects of this type. By default - DownloadAndUpload. If it does not contain the Download flag, changes will not be applied in bpm'online. If it does not contain the Upload flag, new entities will not be selected from bpm'online. |
| FetchColumnNames | String[] | The names of the columns that will be loaded from the local storage. If the value is not specified, it will load all object columns |
DetailEntityConfig class
Synchronization element detail entities settings. Class properties are displayed on Table 11.
Table 11. DetailEntityConfig class properties
| Parameter | Type | Description |
|---|---|---|
| PrimarySchemaName | String | Bpm'online main synchronized entity schema name. |
| ForeignKeyColumnName | String | The column name for the connection with the details of the main object |
LocalItem class
One or more objects from bpm'online that are synchronized with external storage as a unit. It contains a set of SyncEntity class instances. Class properties are shown in Table 12 and methods in Table 13.
Table 12. LocalItem class properties
| Parameter | Type | Description |
|---|---|---|
| Entities | Dictionary> | The SyncEntity collection that is set in accordance with a SyncItem. It contains a collection of "key-value" pairs, where the key is the schema name, and the value is the SyncEntity collection of the scheme. |
| Version | DateTime | The highest value of the date and time of the modification of all Entities in LocalItem. |
| Schema | SyncItemSchema | Synchronization element entity settings schema. |
Table 13. LocalItem class methods
| Method | Returning value type | Description |
|---|---|---|
| AddOrReplace(string schemaName, SyncEntity syncEntity) | Void | Adds new SyncEntity to the collection. If SyncEntity with EntityId already exists, it replaces it. |
| Add(UserConnection userConnection, string schemaName) | SyncEntity | Creates and adds a new SyncEntity collection. |
Synchronization example
An activity and participants are synchronized into one Google-calendar task. An activity (Activity) and participants (SyncEntity) are one element of the synchronization - LocalItem.
RemoteItem - Google task received outside bpm'online. LocalItem - a set of objects (SyncEntity), to which the Google task is converted.
The synchronization schema is displayed on Fig. 1.
Fig. 1. Synchronization schema
