/
Decrease the number of parallel checks (file check concurrency)

Decrease the number of parallel checks (file check concurrency)

Introduction and Increasing Parallelism

The number of concurrently running checks (for example, antivirus engines) is 1 by default. All checks therefore run serially and not in parallel. Since version 2.5.9, this can be easily changed and parallelism can be increased using the command:

sofie file-check-concurrency <number>

However, after increasing it, this number can no longer be reduced so easily. (The Kafka queue used does not support reducing the number of partitions, only increasing it.) The recommendation is therefore to increase it carefully so that there is never a need to reduce it.

If we still get into a situation where reduction must be performed, the detailed procedure is described below.

Reducing Parallelism

This action must be performed when the application is “idle” and no tasks or checks are running. At a minimum, check in the admin interface that Packages → Queued is empty.

The procedure for reducing it is as follows:

Stop the application:

sofie stop

Check that consumers are no longer connected to the queue:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --all-groups

This may take a while, so wait until you see:

Consumer group 'worker' has no active members.

Example:

[root@sofie]# kafka-consumer-groups --bootstrap-server localhost:9092 --describe --all-groups Consumer group 'worker' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID worker syslog 0 1164 1164 0 - - - worker file_workflow 0 116351 116351 0 - - - worker check_integrity 0 1 1 0 - - - worker scanner_status 0 11332 11332 0 - - - worker file_check 5 3670 3670 0 - - - worker file_check 3 29020 29020 0 - - - worker file_check 4 3669 3669 0 - - - worker file_check 1 29015 29015 0 - - - worker file_check 2 3668 3668 0 - - - worker file_check 0 29064 29064 0 - - - worker package_workflow 0 38950 38950 0 - - - worker encryption 0 84 84 0 - - - worker email_queue 0 32 32 0 - - - [root@sofie]#

In this example, we can also see that the parallelism was set to 6 (based on the number of file_check partitions).

Delete and recreate the topic:

WARNING: This action deletes the Kafka message topic/queue (file_check), and if there are unresolved file checks in it, they will be lost and will not be performed. It is therefore advisable to first make sure that nothing is currently being checked. In the table above, this can be verified by checking that the value in the LAG column is 0. If it is not, we can start the application again, wait for completion, and start the procedure again later.

# delete the topic kafka-topics --delete --topic file_check --bootstrap-server localhost:9092 # make sure it has disappeared kafka-topics --describe --topic file_check --bootstrap-server localhost:9092 # create a new topic with 1 partition kafka-topics --create --topic file_check --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 # make sure again that it has been created kafka-topics --describe --topic file_check --bootstrap-server localhost:9092

Modify the configuration so that it matches the new topic:

sed -i 's/^topic\.FILE_CHECK\.concurrency=.*/topic.FILE_CHECK.concurrency=1/' /etc/sofie/META-INF/microprofile-config.properties

Start the application:

sofie start

This completes the process, and everything runs again with the default parallelism of 1, that is, serially.

Parallelism Higher Than 1

If we want concurrency higher than 1, we can directly modify the commands above, or set it to 1 and then increase it using the SOFiE command. For example, for concurrency 3:

sofie file-check-concurrency 3