Setting up the global search service (version 1.4)

Global search components

Deployed on separate servers:

rabbitmq – message broker.

elasticsearch – search engine.

mysql – database of global search component configuration.

redis – database used for caching and speed.

Deployed on a single server:

gs-web-api – web-service for global search component configuration.

gs-scheduler – scheduler of data indexing from BPM into ElasticSearch.

gs-worker – component of data indexing from BPM into ElasticSearch as per the scheduler tasks.

gs-worker-replay – component processing indexing results (gs-worker operation results).

gs-worker-single — a component for targeted indexing of business process data in ElasticSearch on request from the business process.

gs-worker-single-replay – component processing indexing results (gs-worker operation results).

gs-worker-single-task — component for scheduling tasks for gs-worker-single.

To set up the components, download the source files. Download files.

Attention

To create a productive environment, we recommend deploying the ElasticSearch, RabbitMQ and MySQL components in fault-tolerant clusters. If the listed components are deployed separately, disable the deployment of these components in the Docker container via the RUN_RABBITMQ, RUN_ELASTICSEARCH and RUN_MYSQL variables.

Note

You can deploy all global search components on a single server for demonstration purposes. To do this, To do this, in the linux / onsite-custom-env file, set the RUN_RABBITMQ, RUN_ELASTICSEARCH, RUN_MYSQL variable values  to "1".

List of ports used by global search components

Global search setup procedure

1.Install Docker on a physical or virtual machine with Linux OS. Read more >>>

2.Install ElasticSearch. Read more >>>

3.Install RabbitMQ. Read more >>>

4.Set up MySQL. Read more >>>

5.Set up the container variables. Read more >>>

6.Install and run the Global Search Service components. Read more >>>

7.Enable the global search function in bpm’online. Read more >>>

8.Populate the database with initial data. Read more >>>.

Note

Skip steps 2-4 for demo applications with all global search components installed on a single server.

Contents

•Docker setup

•Installing RabbitMQ

•Installing ElasticSearch

•Installing MySQL

•Setting up container variables

•Running containers with the Global Search Service components

•Connecting global search service to bpm’online

•Initial population of the database

Docker setup

Install Docker to Linux OS to deploy global search components. The installation is covered in the Docker documentation. Read more >>>

Run the docker --version command on a Linux machine to verify the installed Docker version.

Installing RabbitMQ

Instructions on RabbitMQ cluster setup for Docker, are available in a separate guide. Read more >>>

Instructions on RabbitMQ cluster setup without Docker, are available in RabbitMQ guide documentation. Read more >>>

Installing ElasticSearch

1.Install ElasticSearch version 5.6.8. See ElasticSearch documentation for more information on the setup procedure. Read more >>>

2.After installing ElasticSearch, install the plugin for morphology search. Download the current version of the Morphological Analysis Plugin for ElasticSearch using the following link. Download the plugin. You can find the morphology plugin installation instructions in the ElasticSearch guide. Read more >>>

3.Add the following command to ElasticSearch docker file to install the plugin:

RUN bin/elasticsearch-plugin install http://dl.bintray.com/content/imotov/elasticsearch-plugins/org/elasticsearch/elasticsearch-analysis-morphology/5.6.8/elasticsearch-analysis-morphology-5.6.8.zip

Installing MySQL

Instructions on MySQL cluster installation for Docker are available in the Docker documentation. Read more >>>

To install the MySQL cluster without Docker, use the instructions in the MySQL guide. Read more >>>

Setting up container variables

All global search component containers are configured via a file with environment variables. The variables are contained in the Docker/linux/onsite-custom-env base file. Edit this file to set the values of variables,

Attention

During the update of global search containers, set the value of "1" for the CLEAR_ELASTICSEARCH_DATA, CLEAR_RABBITMQ_DATA, CLEAR_MYSQL_DATA variables in the Docker/linux/onsite-custom-env file.

Set the values of the below listed parameters only if the RabbitMQ, ElasticSearch or MySQL components are deployed separately.

Additional variables that control the data indexing parameters in ElasticSearch

Running containers with the Global Search Service components

Attention

For the correct container operation, the UTC-time on linux-machine with Docker installed should correspond to the UTC-time on bpm’online DB server. Permissible deviation is up to five minutes. Otherwise, the global search may not index all records.

1.Copy the Docker folder from the application setup files to the linux machine.

2.Run the command.

• For bpm’online applications, deployed on MS SQL:

sh Docker/linux/run.sh mysql mssql 1.4 onsite

• For bpm’online applications, deployed on Oracle:

sh Docker/linux/run.sh mysql oracle 1.0

• For bpm’online applications, deployed on PostgreSQL:

sh Docker/linux/run.sh mysql postgre 1.4 onsite

Note

The "Error response from daemon: network with name net1 already exists” error may occur after restarting the sh Docker/linux/run.sh command, because the docker network named “net1” was created the at the time the command was run. This message is shown because the docker network with the “net1” name was created when you first started the command.

The following Docker volumes will automatically be mounted upon successful container running:

• es1, if RUN_ELASTICSEARCH=1 (es-node1);

• es2, if RUN_ELASTICSEARCH=1 (es-node2);

• mysql, if RUN_MYSQL=1;

• rabbitmq, if RUN_RABBITMQ=1.

Note

Mounting inside the global search containers is performed via the following path: /usr/share/globalsearch.
The “docker volume list” command displays the information about created volumes.
If one of the containers (MySQL, RabbitMQ or ElasticSearch) stops working, the data will be stored on external disk outside the Docker-container. You can learn more about volumes in Docker guide. Read more >>>

Verifying successful running of containers

Enter the docker ps -a command in the console to see all the containers that have been run.

The following containers should be run:

• gs-web-api;

• gs-web-indexing-service;

• gs-scheduler;

• gs-worker:

• gs-worker1;

• gs-worker2;

• gs-worker3;

• gs-worker-replay;

• gs-worker-single;

• gs-worker-single-replay;

• gs-worker-single-task;

• ES, if RUN_ELASTICSEARCH = 1:

• es-node1;

• es-node2;

• rabbitmq, if RUN_RABBITMQ = 1;

• gs-mysql, if RUN_MYSQL = 1;

• gs-redis, if RUN_REDIS=1.

Logging

By default logging is performed during the “stdout” container command execution.

Note

docker logs --tail 100 gs-worker displays 100 last log strings from the gs-worker container.

Note

When you first run the global search components you may receive an error about the failed mysql or rabbitmq connection attempt in their logs. At startup, it is possible that mysql or rabbitmq containers are temporarily unavailable. In this case, continue the connection attempts until the message about the successful connection and start of the container is displayed in stdout: “Now listening on: http://[::]:80 Application started. Press Ctrl+C to shut down”.

Connecting global search service to bpm’online

Actions on the server

To connect global search to bpm'online, perform the following steps on the server:

1.Install the curl utility for http queries.

apt-get install curl

2.Perform the AddSite API-operation and specify:

• [site name] – bpm’online database name, for example, SalesTeamENU;

• [gs-web-api] – external address of the gs-web-api container that was run earlier.

curl -v -X POST -d '{"ApiKey": "testKey", "SiteName": "[site name]"}' -H "Content-Type: application/json" http://[gs-web-api]:81/addsite

3.Perform the AddSearch API-operation and specify:

• [site name] – bpm’online database name, for example, SalesTeamENU;

• [gs-web-api] – external address of the gs-web-api container that was run earlier.

curl -v -X POST -d '{"ApiKey": "testKey", "SiteName": "[site name]"}' -H "Content-Type: application/json" http://gs-web-api:81/addsearch

Note

The indicated query will return URL to the index created in ElasticSearch. Save the URL and use it in the system setting installation SQL-script below.

Settings on the bpm’online side

For MS SQL DBMS:

1.Enable the Feature Toggle function of the global search (GlobalSearch, GlobalSearch_V2) by running the following SQL script:

DECLARE @GlobalSearchFeature NVARCHAR(50) = 'GlobalSearch';
DECLARE @GlobalSearchFeatureId UNIQUEIDENTIFIER = (SELECT TOP 1 Id
FROM Feature WHERE Code = @GlobalSearchFeature);
DECLARE @GlobalSearchV2Feature NVARCHAR(50) = 'GlobalSearch_V2';
DECLARE @GlobalSearchV2FeatureId UNIQUEIDENTIFIER = (SELECT TOP 1 Id
FROM Feature WHERE Code = @GlobalSearchV2Feature);
DECLARE @allEmployeesId UNIQUEIDENTIFIER = 'A29A3BA5-4B0D-DE11-9A51-005056C00008';

IF (@GlobalSearchFeatureId IS NOT NULL)
BEGIN
   IF EXISTS (SELECT * FROM AdminUnitFeatureState WHERE FeatureId = @GlobalSearchFeatureId)
     UPDATE AdminUnitFeatureState SET FeatureState = 1 WHERE FeatureId = @GlobalSearchFeatureId
   ELSE
    INSERT INTO AdminUnitFeatureState (SysAdminUnitId, FeatureState,
 FeatureId) VALUES (@allEmployeesId, '1', @Bulk_ES_DD_FeatureId)
END
ELSE
BEGIN
  SET @GlobalSearchFeatureId = NEWID()
  INSERT INTO Feature (Id, Name, Code) VALUES
(@GlobalSearchFeatureId, @GlobalSearchFeature, @GlobalSearchFeature)
  INSERT INTO AdminUnitFeatureState (SysAdminUnitId, FeatureState,
 FeatureId) VALUES (@allEmployeesId, '1',
@GlobalSearchFeatureId)
END

IF (@GlobalSearchV2FeatureId IS NOT NULL)
BEGIN
   IF EXISTS (SELECT * FROM AdminUnitFeatureState WHERE FeatureId = @GlobalSearchV2FeatureId)
     UPDATE AdminUnitFeatureState SET FeatureState = 1 WHERE FeatureId = @GlobalSearchV2FeatureId
   ELSE
     INSERT INTO AdminUnitFeatureState (SysAdminUnitId, FeatureState, FeatureId) VALUES (@allEmployeesId, '1', @GlobalSearchV2FeatureId)
END
ELSE
BEGIN
  SET @GlobalSearchV2FeatureId = NEWID()
   INSERT INTO Feature (Id, Name, Code) VALUES (@GlobalSearchV2FeatureId, @GlobalSearchV2Feature, @GlobalSearchV2Feature)
   INSERT INTO AdminUnitFeatureState (SysAdminUnitId, FeatureState, FeatureId) VALUES (@allEmployeesId, '1', @GlobalSearchV2FeatureId)
END

2.Run the following script to install the system settings (GlobalSearchUrl, GlobalSearchConfigServiceURL and GlobalSearchIndexingApiUrl):

UPDATE SysSettingsValue
SET TextValue = [specify URL to the ElasticSearch index, string of the following type - http://external.elasticsearch:9200/indexname]
WHERE SysSettingsId = (SELECT TOP 1 Id FROM SysSettings WHERE Code = 'GlobalSearchUrl')

UPDATE SysSettingsValue
SET TextValue = [specify URL to the Global Search Service, string of the following type - http://gs-web-api:81]
WHERE SysSettingsId = (SELECT TOP 1 Id FROM SysSettings WHERE Code ='GlobalSearchConfigServiceUrl')

UPDATE SysSettingsValue
SET TextValue = [specify URL to the Global Indexing Service, string of the following type - http://gs-web-indexing-service:82]
WHERE SysSettingsId = (SELECT TOP 1 Id FROM SysSettings WHERE Code ='GlobalSearchConfigServiceUrl')

3.Restart bpm’online, clear Redis and log into the application.

For Oracle DBMS:

1.Enable the Feature Toggle function of the global search (GlobalSearch, GlobalSearch_V2) by running the following SQL script:

CREATE OR REPLACE FUNCTION
generate_uuid return varchar2 is
   v_uuid varchar2(38);
   v_guid varchar2(32);
BEGIN
   v_guid := sys_guid();
   v_uuid := lower(
                '{' ||
                substr(v_guid, 1,8) || '-' ||
                substr(v_guid, 9,4) || '-' ||
                substr(v_guid, 13,4) || '-' ||
                substr(v_guid, 17,4) || '-' ||
               substr(v_guid, 21) ||
                '}'
               );
   RETURN v_uuid;
END;
/

DECLARE
  GlobalSearchFeature VARCHAR(50) := 'GlobalSearch';
 GlobalSearchFeatureId VARCHAR(38) := NULL;
 GlobalSearchFeatureId_GUID VARCHAR(38) := generate_uuid();
  GlobalSearchV2Feature VARCHAR(50) := 'GlobalSearch_V2';
 GlobalSearchV2FeatureId VARCHAR(38) := NULL;
 GlobalSearchV2FeatureId_GUID VARCHAR(38) := generate_uuid();
  allEmployeesId VARCHAR(38) := '{7F3B869F-34F3-4F20-AB4D-7480A5FDF647}';
 State_GlobalSearch VARCHAR(1) := NULL;
 State_GlobalSearchV2 VARCHAR(1) := NULL;
BEGIN
  SELECT MAX("Id") INTO GlobalSearchFeatureId FROM "Feature" WHERE "Code" = GlobalSearchFeature AND rownum = 1;
  SELECT MAX("Id") INTO GlobalSearchV2FeatureId FROM "Feature" WHERE "Code" = GlobalSearchV2Feature AND rownum = 1;
  SELECT MAX("FeatureState") INTO State_GlobalSearch FROM "AdminUnitFeatureState" WHERE "FeatureId" = GlobalSearchFeatureId AND rownum = 1;
  SELECT MAX("FeatureState") INTO State_GlobalSearchV2 FROM "AdminUnitFeatureState" WHERE "FeatureId" = GlobalSearchV2FeatureId AND rownum = 1;

 IF (GlobalSearchFeatureId IS NULL) THEN
      INSERT INTO "Feature" ("Id", "Name", "Code") VALUES (GlobalSearchFeatureId_GUID, GlobalSearchFeature, GlobalSearchFeature);
      INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchFeatureId_GUID);
   ELSE
     IF (State_GlobalSearch IS NOT NULL) THEN
          UPDATE "AdminUnitFeatureState" SET "FeatureState" = 1 WHERE "FeatureId" = GlobalSearchFeatureId;
       ELSE
          INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchFeatureId_GUID);
     END IF;
 END IF;

 IF (GlobalSearchV2FeatureId IS NULL) THEN
      INSERT INTO "Feature" ("Id", "Name", "Code") VALUES (GlobalSearchV2FeatureId_GUID, GlobalSearchV2Feature, GlobalSearchV2Feature);
      INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchV2FeatureId_GUID);
   ELSE
     IF (State_GlobalSearchV2 IS NOT NULL) THEN
          UPDATE "AdminUnitFeatureState" SET "FeatureState" = 1 WHERE "FeatureId" = GlobalSearchV2FeatureId;
       ELSE
          INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchV2FeatureId_GUID);
     END IF;
 END IF;

END;

2.Run the following script to install the system settings (GlobalSearchUrl, GlobalSearchConfigServiceURL and GlobalSearchIndexingApiUrl):

DECLARE
 URL_SETTING_ID VARCHAR(38) := NULL;
 CONFIG_URL_SETTING_ID VARCHAR(38) := NULL;
 IND_API_SETTING_ID VARCHAR(38) := NULL;

 URL_VAL_ID VARCHAR(38) := NULL;
 CONFIG_URL_VAL_ID VARCHAR(38) := NULL;
 IND_API_VAL_ID VARCHAR(38) := NULL;

  SYS_ADMIN_UID VARCHAR(38) := '{A29A3BA5-4B0D-DE11-9A51-005056C00008}';

  ES_IND VARCHAR(500) := '[enter the URL to the ElasticSearch index, string of the following type - http://external.elasticsearch:9200/indexname]';
  CONFIG_URL VARCHAR(500) := '[enter the URL to the Global Search Service, string of the following type - http://gs-web-api:81]';
  IND_API_URL VARCHAR (500): = '[enter the URL to the Global Search Indexing Service, string of the following type - http://gs-web-indexing-service: 82]’
BEGIN
  SELECT "Id" INTO URL_SETTING_ID FROM "SysSettings" WHERE "Code" = 'GlobalSearchUrl';
  SELECT "Id" INTO CONFIG_URL_SETTING_ID FROM "SysSettings" WHERE "Code" = 'GlobalSearchConfigServiceUrl';
  SELECT "Id" INTO IND_API_SETTING_ID FROM "SysSettings" WHERE "Code" = 'GlobalSearchIndexingApiUrl';

  SELECT MAX("Id") INTO URL_VAL_ID  FROM "SysSettingsValue" WHERE "SysSettingsId" = URL_SETTING_ID;
  SELECT MAX("Id") INTO CONFIG_URL_VAL_ID  FROM "SysSettingsValue" WHERE "SysSettingsId" = CONFIG_URL_SETTING_ID;
  SELECT MAX("Id") INTO IND_API_VAL_ID  FROM "SysSettingsValue" WHERE "SysSettingsId" = IND_API_SETTING_ID;

 IF (URL_VAL_ID IS NULL)
   THEN
     INSERT INTO "SysSettingsValue"
       ("SysSettingsId", "SysAdminUnitId", "IsDef", "TextValue")
       VALUES
        (URL_SETTING_ID, SYS_ADMIN_UID, '1', ES_IND);
   ELSE
      UPDATE "SysSettingsValue" SET "TextValue" = ES_IND WHERE "SysSettingsId" = URL_SETTING_ID;
 END IF;

 IF (CONFIG_URL_VAL_ID IS NULL)
   THEN
     INSERT INTO "SysSettingsValue"
       ("SysSettingsId", "SysAdminUnitId", "IsDef", "TextValue")
       VALUES
        (CONFIG_URL_SETTING_ID, SYS_ADMIN_UID, '1', CONFIG_URL);
   ELSE
      UPDATE "SysSettingsValue" SET "TextValue" = CONFIG_URL WHERE "SysSettingsId" = CONFIG_URL_SETTING_ID;
 END IF;

 IF (IND_API_VAL_ID IS NULL)
   THEN
     INSERT INTO "SysSettingsValue"
       ("SysSettingsId", "SysAdminUnitId", "IsDef", "TextValue")
       VALUES
        (IND_API_SETTING_ID, SYS_ADMIN_UID, '1', IND_API_URL);
   ELSE
      UPDATE "SysSettingsValue" SET "TextValue" = IND_API_URL WHERE "SysSettingsId" = IND_API_SETTING_ID;
 END IF;
END;

3.Restart the bpm’online application, clear Redis and log into the application.

For PostgreSql DBMS:

1.Enable the Feature Toggle function of the global search (GlobalSearch, GlobalSearch_V2) by running the following SQL script:

DO $$

DECLARE
    GlobalSearchFeature VARCHAR(50) := 'GlobalSearch';
   GlobalSearchFeatureId uuid;
    GlobalSearchV2Feature VARCHAR(50) := 'GlobalSearch_V2';
   GlobalSearchV2FeatureId uuid;
    allEmployeesId uuid := 'A29A3BA5-4B0D-DE11-9A51-005056C00008';

BEGIN

  SELECT "Id" INTO GlobalSearchFeatureId FROM "Feature"
  WHERE "Code" = GlobalSearchFeature
  LIMIT 1;
  IF (GlobalSearchFeatureId IS NOT NULL)
     THEN
          IF EXISTS (SELECT * FROM "AdminUnitFeatureState" WHERE "FeatureId" = GlobalSearchFeatureId) THEN
             UPDATE "AdminUnitFeatureState" SET "FeatureState" = 1 WHERE "FeatureId" = GlobalSearchFeatureId;
         ELSE
              INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchFeatureId);
         END IF;
  ELSE
      GlobalSearchFeatureId := uuid_generate_v4();
       INSERT INTO "Feature" ("Id", "Name", "Code") VALUES (GlobalSearchFeatureId, GlobalSearchFeature, GlobalSearchFeature);
       INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchFeatureId);
  END IF;

  SELECT "Id" INTO GlobalSearchV2FeatureId FROM "Feature”
  WHERE "Code" = GlobalSearchV2Feature
  LIMIT 1;
  IF (GlobalSearchV2FeatureId IS NOT NULL)
   THEN
        IF EXISTS (SELECT * FROM "AdminUnitFeatureState" WHERE "FeatureId" = GlobalSearchV2FeatureId) THEN
            UPDATE "AdminUnitFeatureState" SET "FeatureState" = 1 WHERE "FeatureId" = GlobalSearchV2FeatureId;
       ELSE
           INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchV2FeatureId);
       END IF;
  ELSE
      GlobalSearchV2FeatureId := uuid_generate_v4();
       INSERT INTO "Feature" ("Id", "Name", "Code") VALUES (GlobalSearchV2FeatureId, GlobalSearchV2Feature, GlobalSearchV2Feature);
       INSERT INTO "AdminUnitFeatureState" ("SysAdminUnitId", "FeatureState", "FeatureId") VALUES (allEmployeesId, '1', GlobalSearchV2FeatureId);
  END IF;
END $$;

2.Run the following script to install the system settings (GlobalSearchUrl, GlobalSearchConfigServiceURL and GlobalSearchIndexingApiUrl):

UPDATE "SysSettingsValue"
SET TextValue = [specify URL to the ElasticSearch index, string of the following type - http://external.elasticsearch:9200/indexname]
WHERE "SysSettingsId" = (SELECT "Id" FROM "SysSettings" WHERE "Code" =
 'GlobalSearchUrl' LIMIT 1 );

UPDATE "SysSettingsValue"
SET TextValue = [specify URL to the Global Search Service, string of the following type - http://gs-web-api:81]
WHERE "SysSettingsId" = (SELECT "Id" FROM "SysSettings" WHERE "Code" = 'GlobalSearchConfigServiceUrl' LIMIT 1 );

UPDATE "SysSettingsValue"
SET TextValue = [specify URL to the Global Indexing Service, string of the following type - http://gs-web-indexing-service:82]
WHERE "SysSettingsId" = (SELECT "Id" FROM "SysSettings" WHERE "Code" = 'GlobalSearchIndexingApiUrl' LIMIT 1 );

3.Restart bpm’online, clear Redis and log into the application.

Initial population of the database

The database is populated with system records once the GSS is launched. Prior to that, there is not even a database file, since the database is automatically generated.

See also

•API global search service description

•Global search