Additional setup
PDF

Application server web-farm

Products
All Creatio products

You can enhance the performance of large-scale Creatio projects (up to several thousand users) through horizontal scaling, i.e., increasing the number of servers with deployed Creatio applications and distributing the workload between them.

The load balancer may be either hardware or software. To work in fault-tolerant mode, use the HTTP/HTTPS traffic balancer that supports the WebSocket protocol. Creatio has been tested on HAProxy software load balancer. There are cases of successful implementation of other balancers, e.g., Citrix, Cisco, NginX, FortiGate, MS ARR.

This guide covers horizontal scaling of Creatio application using a free open-source load balancer (HAProxy), designed for distributing the load between several application servers.

General deployment procedure

.NET Framework application servers

To deploy Creatio with horizontal scaling of .NET Framework application servers:

The “value” number should be the same for all application instances of the web-farm.

Attention. Starting with Creatio version 7.14.1, the <add key="TenantId" value="...”/> key can only be added to the internal Web.config file (Terrasoft.WebApp\Web.config). Adding the key to an external Web.config file may lead to application failures.

  1. Deploy all needed Creatio application instances in a web-farm.

    Note. It is recommended to specify identical names in IIS and the Application pool setting for all instances of the application.

  2. Specify the same SQL and Redis databases in the ConnectionStrings.config file for all instances.

    <add name="redis" connectionString="host=DOMAIN.COM;db=0;port=6379;maxReadPoolSize=10;maxWritePoolSize=500"/>
    <add name="db" connectionString="Data Source=DOMAIN.COM;Initial Catalog=DatabaseName;Integrated Security=SSPI; MultipleActiveResultSets=True;Pooling-true;Max Pool Size=100"/>
  3. In the <appSettings> block of the application’s “Web.config” file, add the following key:

    <add key="TenantId" value="1" />
  4. Generate a unique machineKey value for one of the application instances. Read more about it in the Modify Web.config article. Copy the resulting value and specify it in the Web.config files for each application instance. You can locate the files in the root Creatio folder and the Terrasoft.WebApp folder.

  5. Grant access permissions to created application directories for the IUSR user and the user who launches the Application pool in IIS.

  6. Set up a load balancer (e.g., HAProxy) for distributing the workload between the deployed application servers.

  7. If necessary, set up workload balancing for database and session servers.

    Note. More information about the clustering setup is available in the MSSQL and Oracle user guides. The fault tolerance setup using Redis Sentinel is described in the “Redis Sentinel” article.

.NET Core application servers

To deploy Creatio with horizontal scaling of .NET Core application servers:

  1. Deploy the needed number of Creatio application instances.
  2. Specify the same SQL and Redis databases in the ConnectionStrings.config file for all instances.
  3. Navigate to the root application directory of any application instances (the root directory contains the Terrasoft.WebHost.dll file).
  4. Run the following command:
    dotnet Terrasoft.WebHost.dll configureWebFarmMode

    As a result, the configuration files of the current application instance will be updated.

  5. Copy all configuration files of the current application instance to the root folders of the other application instances.
  6. Set up a load balancer (e.g., HAProxy) for distributing the workload between the deployed application servers.
  7. If necessary, set up workload balancing for database and session servers.

Note. More information about the clustering setup is available in the documentation provided by the vendor of the DBMS. The fault tolerance setup using Redis Sentinel is described in the “Redis Sentinel” article.

Install HAProxy balancer

The HAProxy load balancer supports a range of free open-source OS. In this guide, we will cover one of the simpler methods of deploying HAProxy on the Debian OS via the haproxy.debian.net service.

  1. Open the installation service page by clicking https://haproxy.debian.net/.

  2. Select the OS and its version, afterward select the HAProxy version.

    Note. Use the cat /etc/issue command to check the version of the currently installed Debian OS.

    As a result, the service will generate a set of commands that must be executed in the Debian OS to install HAProxy.

    Fig. 1 Example of HAProxy installation commands generated by the haproxy.debian.net service
    haproxy_settings_commands.png
  3. Execute the generated commands one after another.

Set up HAProxy balancer

To set up HAProxy, modify the haproxy.cfg file. You can find the file under the following path:

.../etc/haproxy/haproxy.cfg

Primary setup (required)

Add two sections necessary for HAProxy operation: frontend and backend.

The frontend section

Add two settings to the frontend section: bind and default_backend.

  • In the bind setting, specify the address and the port that will receive requests distributed by HAProxy.

  • In the default_backend option, specify the name that will match the name of the backend section.

As a result, the setup will look as follows:

frontend front
maxconn 10000
#Using these ports for binding
bind *:80
bind *:443
#Convert cookies to be secure
rspirep ^(set-cookie:.*) \1;\ Secure
default_backend creatio

The backend section

Add 2 required settings to the backend section:

  • In the balance parameter, specify the type of balancing, e.g., roundrobin. More information about different types of balancing is available in HAProxy documentation.

  • Use the server parameter to specify all servers (or “nodes”) for distributing the load.

Add a separate “server” parameter for each server (i.e. the deployed Creatio application instance) and specify the server/port addresses and weight. The server weight allows the balancer to distribute the load based on the physical capabilities of the servers. The more weight is specified for the server, the more requests it receives. For example, if you need to distribute the load between 2 Creatio application servers, add 2 “server” parameters to the backend:

server node_1 server address:port weight
server node_2 server address:port weight

As a result, the setup will look as follows:

backend creatio
#set balance type
balance roundrobin

server node_1 nodeserver1:80 check inter 10000 weight 2
server node_2 nodeserver2:80/sitename check inter 10000 weight 1

The new settings will be applied as soon as you restart HAProxy. To restart HAProxy, use the following command:

service haproxy restart

Check server status

The HAProxy balancer works with the following server statuses:

Status

Status details

UP

The server is operational.

UP - transitionally DOWN

The server registers as treated as operational at the moment, but the last health check has failed. As a result, the server is currently switching to the DOWN status.

DOWN - transitionally UP

The server registers as treated as not operational at the moment, but the last health check has succeeded. As a result, the server is currently switching to the UP status.

DOWN

The server is not operational.

 

Health checks initiate changes in a server’s operational status. The simplest health check requires adding the “check” keyword in the server setup string. Running the health check requires the server’s IP and TCP port. Health check example:

server node1 ... check
option httpchk GET /Login/NuiLogin.aspx
option httpchk GET /0/ping

Set up web statistics (optional)

To enable web-statistics, add a new “listen” section with the following parameters: bind, mode http, stats enable, stats uri. The syntax is as follows:

listen stats # Define a listen section called "stats"
          bind :9000 # Listen on localhost:9000
          mode http
          stats enable  # Enable stats page
          stats uri /haproxy_stats  # Stats URI

As a result, the web statistics of Creatio load balancing will be available for viewing in the browser.

Fig. 1 Example of a load balancer web statistics
load_balancer_stats.png

To view the statistics, follow the path: balancer address:9000/haproxy_stats.

Make the users' IP addresses visible in the audit log for .NET Core (optional)

With a web-farm, user requests reach the servers through a load balancer and/or a proxy server. As such, by default, the audit log displays the IP address of the proxy that forwarded the request last, not the user's actual IP address.
You can configure the audit log to display the user's actual IP address. To do this:

  1. Configure the balancer so that each request it forwards to one of the Creatio application instances has a header with "ForwardedForHeaderName" name and the user's IP address value.
  2. Configure the Creatio application instances accordingly.
    1. Navigate to the instance's root directory and open appsettings.json.
    2. Edit the "ForwardedHeaders" section so that it reads:
      {
           ...
           "ForwardedHeaders": {
               "Enable": true,
               "ForwardedForHeaderName": "X-Forwarded-For",
               "KnownProxiesIP": [trusted IP addresses]
           }
           ...
      }

      Where:
      "Enable" toggles the Forwarded headers processing function in the web application
      "ForwardedForHeaderName" is the name of the header that contains the IP address
      "KnownProxiesIP" is the trusted IP address list. Creatio will process the "ForwardedHeader" value only if it receives a request from these IPs. They may belong to the load balancer, reverse proxy, etc. If you leave this value empty, Creatio will process the "ForwardedHeader" value received from any IP address.

    3. Repeat steps a–b for all Creatio application instances in your web-farm.

Case
"KnownProxiesIP": ["127.0.0.1", "12.34.56.78", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"]