Repositories. Types and recommendations on use

Introduction

In Creatio, business data storage is implemented as a relational database (MS SQL, Oracle or PostgreSQL). The data is used for solutions of different user tasks, execution of business processes, etc. The application ensures execution of the following operations:

1. Storage of user and application data (user profile, session data etc.).
2. Data communication between web farms.
3. Intermediate data storage at the time of restart of application or webfarm.
4. Execution of intermediate actions with data before their placing in repository.

For the purpose of these tasks, data repository technology is implemented in Creatio’s architecture. The object model of classes, being a unified API for access from application to data, located in an external repository (currently Redis is used in Creatio as the external repository) forms the base of this technology.

Creatio repositories focus on execution of service functions or arrangements of data handling but they also can be used for solving user tasks (in configuration business logic).

Redis as a Creatio repository server

Creatio repository server is represented by Redis – a high-efficient non-relational data repository.

The data model of Redis is based on "key-object" pairs. Redis supports access only with a unique key and can be successfully used for storage of serialized objects. Data, placed into the repository, is stored as binary serialized objects in Creatio.

Redis supports several data storage strategies:

• Storage of data only in memory. Persistent database is converted to caching server.
• Periodical saving of data to disc (by default) – periodical copy saving (snapshot) (once per 1-15 minutes depending on time for creation of previous copy and number of changed keys).
• Transaction log. A synchronous record of each change in special append-only log-file.
• Replication. You can assign a primary to each server. Then all changes in master will be reproduced in the replica.

The definite data storage method is determined by configuring the Redis server. In Creatio (at present), data is stored in memory with periodic saving of data to disc but data replication function is not supported.

For more details on Redis see official documentation.

Creatio repository type. Data storage and cache.

Creatio supports two types of repositories, i.e. data and cache repositories.

A data repository is designed for intermediate storage of rarely modified "long-term" data. A cache repository stores operation data.

Individual logical levels of data placement are additionally determined for each type of repository (table 1, 2).

Table 1. Data repository levels

Table 2. Cache levels

Such differentiation of repositories is implemented for the purpose of logical separation of units in data repository and convenience of further use of the source code. Additionally, such separation creates solutions for the following tasks:

• Isolation of data between workspaces and user sessions
• Conditional classification of data
• Control of the data life cycle

All repository and cache data can be located physically on an abstract data storage server. The exception is the data of the Request level repository that are stored directly in the memory.

At present, the Creatio repository server is Redis. In common cases, it may be a user repository, accessed through unified interfaces. It is necessary to take into account the fact that repository access operations are resource-intensive since they are associated with serialization / deserialization of data and network communication.

The data repository and cache represent the following possibilities for data handling:

• Access to data by key for reading / recoding
• Deletion of data from repository by key

The key difference between a data repository and cache is the control of the object life cycle within them.

Data will be stored in the storage till the explicit deletion. The life cycle of such objects is limited by query execution time (for data of Request level repository) or session existence time (for data of Session level repository).

There is such a notion as aging time for data. It determines time limit of validity of cache items. All items are deleted from cache, regardless of aging time, in the following cases:

• Session ending (items of cache and data repository of Session level)
• Implicit deletion of workspace (items of Workspace level cache)

The items of the Application level cache are stored for the entire period of existence of the application and can be deleted only by clearing the external repository.

Object model of Creatio repositories

Creatio classes and interfaces for data repository and cache operations

Creatio uses a series of classes and interfaces (located in Terrassot.Core.Store name space) to work with the database and cache (“ .NET class libraries of platform core”). The main ones are listed below.

IBaseStore base repository

Base functions of all types of repositories are represented by IBaseStore. Properties and methods of this interface are used to implement the following:

• Access to data by key for reading/recording (this[string key] indexer)
• Deletion of data from the repository by set key (Remove(string key) method)
• Initialization of the repository by set parameter list (Initialize(IDictionary<string, string> parameters) method). At present, parameters for initialization of Creatio repositories are read from configuration file. Parameter lists are set in storeDataAdapter (for data repository) and storeCacheAdapter (for cache) sections. In common cases, parameters can be set randomly.

IDataStore data repository

The interface determines the specifics of data repository operations. It is an inheritor of IBaseStore repository base interface. The interface provides an additional possibility for getting the list of all repository keys (the Keys property).

ICacheStore cache storage

The interface determines the specifics of cache operations. It is an inheritor of IBaseStore repository base interface. The GetValues(IEnumerable keys) property returns cache object dictionary with set keys. This method optimizes repository operation upon simultaneous receiving of data set. This method enables the optimization of working with the repository while simultaneously receiving a dataset.

The Store class

Static class for access to caches and data repositories of different levels. Levels of data repositories and cache are determined in the Terrasoft.Core.Store.DataLevel and CacheLevel enumerations (table 1 and 2).

The Store class has 2 static properties:

• The Data property returns an instance of the data repository provider.
• The Cache property returns an instance of the cache provider.

Example 1 demonstrates working with the cache and the data store using the Store class.

Example 1

// Getting a link to the session-level data repository.
IDataStore dataStore = Store.Data[DataLevel.Session];

// Place the "Data Test Value" value with the "DataKey" key in the data repository.
dataStore["DataKey"] = "Data Test Value";

// Getting a link to the application-level cache of the workspace.
ICacheStore cacheStore = Store.Cache[CacheLevel.Workspace];

// Removing an item with the "CacheKey" key from the cache.
cacheStore.Remove("CacheKey");

Access to data repositiries and cache from UserConnection

Access to the data repositories and application caches from the configuration code can be obtained using the properties of the static Store class of the Terrasoft.Core.Store name space. An alternative way to access the data repository and cache in the configuration logic, which avoids the use of long property names and the connection of additional assemblies, is accessed through an instance of the UserConnection class.

The UserConnection class implements a number of helper properties that enable quick access to data repositories and caches of various levels:

• ApplicationCache returns a link to the application-level cache.
• WorkspaceCache returns a link to the workspace cache.
• SessionCache returns a link to the session-level cache.
• RequestData returns a link to the query-level data repository.
• SessionData returns a link to the session-level data repository.
• ApplicationData returns a link to the application-level data repository.

Example 2

// The key with which the value will be placed in the cache.
string cacheKey = "SomeKey";

// Putting the value into the session-level cache via the UserConnection property.
UserConnection.SessionCache[cacheKey] = "SomeValue";

// Retrieving a value from the cache via the Store property.
// As a result, the variable valueFromCache will contain the value "SomeValue".
string valueFromCache = Store.Cache[CacheLevel.Session][cacheKey] as String;

Using cache in EntitySchemaQuery

EntitySchemaQuery implements a mechanism for working with the repository (Creatio cache or user-defined arbitrary repository). Working with the cache allows you to optimize the efficiency of operations by accessing cached query results without additional access to the data repository. When the EntitySchemaQuery query is executed, the data received from the database on the server is placed in the cache, which is determined by the Cache property with the key that is set by the CacheItemName property. By default, the cache of theEntitySchemaQuery request is the session-level Creatio cache with local storage. In general, the query cache can be an arbitrary repository that implements the ICacheStore interface.

Example 3 shows the process of working with the application cache when executing the EntitySchemaQuery request. The code in the example builds a query that returns a list of all cities in the City schema. When you receive the results of the query (after calling the GetEntityCollection() method), these results are placed in the cache, which can then be used to retrieve the collection of query elements without additional access to the repository.

Example 3

// Creating an instance of the EntitySchemaQuery request with the root City schema.
var esqResult = new EntitySchemaQuery(UserConnection.EntitySchemaManager, "City");

// Add the city name column to the query.
esqResult.AddColumn("Name");

// Identifying the key under which the results of query execution will be stored in the cache.
// Session level cache with local data caching (since Cache property of the object is not determined)
// is used as cache.
esqResult.CacheItemName = "EsqResultItem";

// Execution of query to database for receiving resultant object collection. 
// Query results will be placed in cache after completion of this operation. Upon further access to 
// esqResult for receiving object collection query (if query was not changed) these objects will 
// be taken from session cache.
esqResult.GetEntityCollection(UserConnection);  

Repository and cache proxy classes

Proxy class notion and purpose

Repositories and caches can be accessed either directly (through the Store class properties) or through proxy classes in Creatio.

Proxy classes are special objects that act as intermediate links between repositories and calling code. Proxy classes allow intermediate operations with data before they are read / edited in the repository. Each proxy class is a repository in itself.

Proxy class applications

1. Initial setup and configuration of the application

You can configure the use of proxy classes for data repositories and caches in the Web.config configuration file. The proxy classes for the corresponding data repository or cache are configured in the storeDataAdapter and storeCacheAdapter sections, respectively. They add a proxies section, which lists all the proxy classes that are applied to the repository (example 4).

Example 4

<storeDataAdapters>
    <storeAdapter levelName="Request" type="RequestDataAdapterClassName">
        <proxies>
            <proxy name="RequestDataProxyName1" type="RequestDataProxyClassName1" />
            <proxy name="RequestDataProxyName2" type="RequestDataProxyClassName2" />
            <proxy name="RequestDataProxyName3" type="RequestDataProxyClassName3" />
        </proxies>
    </storeAdapter>
</storeDataAdapters>

<storeCacheAdapters>
    <storeAdapter levelName="Session" type="SessionCacheAdapterClassName">
        <proxies>
            <proxy name="SessionCacheProxyName1" type="SessionCacheProxyClassName1" />
            <proxy name="SessionCacheProxyName2" type="SessionCacheProxyClassName2" />
        </proxies>
    </storeAdapter>
</storeCacheAdapters> 

When the application is downloaded, the settings are read from the configuration file and applied to the corresponding repository type. This enables you to build chains of proxy classes that will be executed sequentially one after another. The order of the proxy classes in the execution chain corresponds to their order in the configuration file. In this case, the proxy class listed last in the proxies section is the first in the chain of execution, i.e. the execution is done "from the bottom up".

Take into account the following:

• The "final point" of the application of proxy classes is a definite cache or data repository, for which this chain is determined.
• A separate proxy class knows the cache or repository it works with. The link to it is determined by ICacheStoreProxy.CacheStore property or IdataStoreProxy.DataStore property. A proxy class doesn't know what exactly this property refers to, i.e. other proxy class, final repository or cache. At the same time, this proxy can act as repository, with which other proxy classes can work.

Thus, according to the settings shown in Example 4, the chain of execution of proxy classes (e.g., data repository) will be as follows: RequestDataProxyName3 > RequestDataProxyName2 > RequestDataProxyName1 > RequestDataAdapterClassName (final query level data repository).

2. Delimiting data of different application users

Isolating data between different application users is done with the help of proxy classes. The simplest solution to this problem is to transform the value keys before placing them in the repository (e.g., by adding an additional prefix to the key that is specific to the user). Using such proxy classes ensures the uniqueness of storage keys. This, in turn, helps to avoid data loss and corruption while different users simultaneously try to write different values ​​in the repository with the same key. An example of working with key transformation proxy classes is shown below. Generally speaking, you can implement as complex logic as you like in proxy classes.

3. Performing other intermediate actions with the data before placing it in the repository.

In proxy classes, you can implement the logic of performing any arbitrary actions with data before placing or getting it from the repository. Putting the logic of processing data into a proxy class helps avoid code duplicates, which in turn facilitates its maintenance.

Basic proxy class interfaces

A class must implement one (or both) of the Terrasoft.Core.Store namespace interfaces in order to be used as a proxy for working with repositories:

• IDataStoreProxy is an interface for proxy classes of the data storage
• ICacheStoreProxy is the interface of cache proxy classes.

Each of these interfaces has one property - a reference to that storage (or cache) with which the given proxy class is working. For the IDataStoreProxy interface, it is the DataStore property. For the ICacheStoreProxy interface, it is the CacheStore property.

Key transformation proxy classes

In Creatio, there is a number of proxy classes that implement the logic of transformation of value keys placed in the repository.

1. The KeyTransformerProxy class

This is an abstract class which acts as a base class for all proxy classes that convert cache keys. It implements the methods and properties of the ICacheStoreProxy interface. When creating custom proxy classes for key transformation, it is recommended to inherit from this class to avoid logic duplicates.

2. The PrefixKeyTransformerProxy

A proxy class that converts cache keys by adding a prefix to them.

Example 5 demonstrates working with the session-level cache via the PrefixKeyTransformerProxy proxy class.

Example 5

// Key, with which a value will be placed in cache through proxy.
string key = "Key";

// Prefix that will be added to value key by proxy class.
string prefix = "customPrefix";

// Creation of proxy class that will be used for recording of values into session level cache.
ICacheStore proxyCache = new PrefixKeyTransformerProxy(prefix, Store.Cache[CacheLevel.Session]);

// Recording value with the "key" key in cache through proxy class. 
// This value is recorded into global cache of session level with "prefix+key" key. 
proxyCache[key] = "CachedValue";

//  Receiving the value on "key" key through proxy.
var valueFromProxyCache = (string)proxyCache[key];

// Receiving the value on "prefix+key" key directly from session level cache.
var valueFromGlobalCache = (string)Store.Cache[CacheLevel.Session][prefix + key]; 

As a result, the valueFromProxyCache and valueFromGlobalCache variables will contain the same value of "CachedValue".

3. The DataStoreKeyTransformerProxy class

A proxy class that converts repository keys by adding a prefix to them.

Example 6 demonstrates working with the repository via the PrefixKeyTransformerProxy proxy class.

Example 6

// Key with which value will be placed into the repository through proxy.
string key = "Key";

// Prefix that will be added to the value key by means of a proxy class.
string prefix = "customPrefix";

// Creation of proxy class that will be used for recording values into session level repository.
IDataStore proxyStorage = new DataStoreKeyTransformerProxy(prefix) { DataStore = Store.Data[DataLevel.Session] };

// Recording a value with the key into the repository through proxy class.
// This value is recorded into global cache of session level with "prefix+key".
proxyStorage[key] = "StoredValue";

// Receiving a value for the key through proxy.
var valueFromProxyStorage = (string)proxyStorage[key];

// Receiving a value on the "prefix+key" key directly from session level cache.
var valueFromGlobalStorage = (string)Store.Data[DataLevel.Session][prefix + key]; 

As a result, the valueFromProxyStorage and valueFromGlobalStorage variables will contain the same value of "StoredValue".

Local data caching

The mechanism of local data caching is implemented on the basis of proxy classes in Creatio. The main purpose of data caching is to reduce the load on the repository server and the time it takes to run queries when dealing with data that is rarely changed.

The logic of the local caching mechanism is implemented by the internal LocalCachingProxy proxy class. This proxy class caches the data on the current web farm node. The class monitors the lifetime of cached objects and receives data from the global cache only if the cached data is not relevant.

The ICacheStore interface from the static CacheStoreUtilities class are used to apply the local caching mechanism:

• WithLocalCaching() is an overloaded method that returns the LocalCachingProxy class instance which executes the local caching.
• WithLocalCatchingOnly(string) is a method that executes the local caching of data for a set group of elements while monitoring their validity.
• ExpireGroup(string) is a method that sets an aging checkbox for a set group of items. All items of the set group became invalid and do not return upon data query upon calling of this method.

Example 7 demonstrates a workspace cache operation with the use of local caching.

Example 7

// Creating the first proxy class that executes local data caching. All items 
// recorded in cache through this proxy refer to Group1 validity control group.
ICacheStore cacheStore1 = Store.Cache[CacheLevel.Workspace].WithLocalCaching("Group1");

// Adding the element to cache through proxy class.
cacheStore1["Key1"] = "Value1";

// Creating the second proxy class that executes local data caching.
// All items recorded in cache through this proxy also refer to Group1 validity control group. 
ICacheStore cacheStore2 = Store.Cache[CacheLevel.Workspace].WithLocalCaching("Group1");
cacheStore2["Key2"] = "Value2";

// Aging checkbox is set for all items of Gtoup1. Items with Key1 and Key2 are considered to be aged
// since they refer to one validity control group, Group1, regardless of the fact that 
//  they are added to cache through different proxies.
cacheStore2.ExpireGroup("Group1");

// The attempt to receive values from cache on Key1 and Key 2 after these items were marked as
// aged. As a result, cachedValue1 and cachedValue 2 variables will contain null value.
var cachedValue1 = cacheStore1["Key1"]; 
var cachedValue2 = cacheStore1["Key2"];

Specific features of database and cache usage

To improve the efficiency of working with data repositories and cache in the configuration logic and in the business processes implementation logic, consider the following:

1. All objects placed in the repository must be serializable. This requirement is due to the specifics of working with the core of the application. If you save data to the repository (except for data repositories of the Request level), the object is pre-serialized, and deserialized when received.

2. Accessing the repository is a relatively resource-intensive operation. The code should avoid unnecessary access to the repository. Examples 8 and 9 provide correct and incorrect versions of the cache.

Example 8

Incorrect:

// Network access and data deserialization are executed.
if (UserConnection.SessionData["SomeKey"] != null)
{
    // Network access and data deserialization are executed repeatedly.
    return (string)UserConnection.SessionData["SomeKey"];
}

Correct (1st option):

// Getting the object from the repository to the intermediate variable.
object value = UserConnection.SessionData["SomeKey"];

// Verification of the intermediate variable value.
if (value != null)
{
    // Value return.
    return (string)value;
}

Correct (2nd option):

// The use of GetValue() extending method.
return UserConnection.SessionData.GetValue<string>("SomeKey");

Example 9

Incorrect:

// Network access and data deserialization are executed.
if (UserConnection.SessionData["SomeKey"] != null)
{
    // Network access and data deserialization are executed repeatedly.
    UserConnection.SessionData.Remove("SomeKey");
}

Correct:

// Deletion is executed immediately without any preliminary verification.
UserConnection.SessionData.Remove("SomeKey");

3. Any changes made to the state of an object obtained from a repository or cache occur locally in memory and are not automatically stored in the repository. Therefore, for these changes to appear in the repository, the modified object must be explicitly written to it (example 10).

Example 10

// Receiving the value dictionary form session data repository on "SomeDictionary" key.
Dictionary<string, string> dic = (Dictionary<string, string>)Store.Data[DataLevel.Session]["SomeDictionary"];

// Change of the dictionary value item. Changes in the repository are not fixed.
dic["Key"] = "ChangedValue";

// Adding a new item to the dictionary. Changes are not fixed in the dictionary.
dic.Add("NewKey", "NewValue");

//  The dictionary is recorded in the data repository on the "SomeDinctionary" key.
// All changes are now fixed in the repository.
Store.Data[DataLevel.Session]["SomeDictionary"] = dic;