Upgrade notes (Instructions for upgrading to a new version)

Contents

General instructions

Regardless of the application version from and to which the update or upgrade is proceeding, the following rules and recommendations apply.

Basic rules

• Regular updates and upgrades are recommended. Especially when a new major version is released.

• It is not recommended to skip major version upgrades and then later perform an upgrade through multiple major versions. Nonetheless if this happens, it is necessary to go through all the specific steps for these versions and perform those in ascending version order, see below.

• It is recommended to perform a backup before an update or upgrade, for example in the form of a VM snapshot, or another appropriate way in which the app is being backed up. In case of an unexpected problem it should be then possible to revert back, using the backup.

Upgrade versus update

Until the version 2.0.6 there was only the update, which always performed an update of the application to the latest released version. Since the version 2.0.6 (including) the process was split in two different steps: update and upgrade.

Update

The update of the application performs an update to the latest minor version in the current version branch (same major version), for example from 2.0.6 to 2.0.8 or from 2.1.1 to 2.1.5. It will no longer do an upgrade between major versions (version branches), for example from 2.0.x to 2.1.x. (This is now true even for older versions, so for example executing update in the version 2.0.1 will just update to the latest 2.0.x and will not jump to next major version 2.1.x or further.)

Upgrade

The upgrade of the application performs an upgrade to a latest version in the next major version branch, if (and only if) one is available. So for example upgrade will perform a transition from the current version 2.0.x to the last version 2.1.x branch. If the next major version branch does not exist, the upgrade will do nothing. It is therefore not possible to use the upgrade to update the app to a new minor version.

How to combine

Perform common regular updates of the application using the update. If a new major version branch is released (eg. 2.1.0), then to stay up to date it is necessary to perform an upgrade. After that just do regular updates again to update to next minor versions in the current branch.

Recommended update or upgrade procedure

Mandatory steps

To update the SOFiE application (to the latest minor version in the current major version branch) run the following command from the command line:

This command will automatically perform:

• backup of the application database

• download and install the rpm packages with the new version

• perform a migration of the database structures (schema update)

• restart of the SOFiE application services

To upgrade the SOFiE application (to the latest version in the next major version branch) run the following command from the command line:

This command will perform an upgrade to a new major version branch, if one exists, and then run an update. If a new major version branch does not exist, nothing is done, not even the update.

Next the version specific steps must be performed for all versions through and up to which the upgrade went through. See the details for each version below.

Since version 2.5.10, the update and upgrade commands do not execute the action immediately, but first interactively ask for confirmation. The prompt can be suppressed and the action started immediately without asking, as before, by adding the -y or --yes parameter.

Optional, but recommended, steps

Before and after the update or upgrade it is recommended to check the current version and status of the application using the following commands:

Besides updating or upgrading the SOFiE application itself, it is also recommended to perform an upgrade of the operating system, which in the RHEL based systems can be done using the command:

Then it is recommended to restart the server, so it can be verified, that after all the updates and upgrades the system and the application will start up correctly when rebooted:

Finally we check, that the application is working as it should. For example by performing an application test by sending a test package to a user and checking that it arrived and that the user can download it. From the administrator’s point of view we check, that all the configured scans for the package were performed and that the audit log does not contain any errors.

Update or upgrade to a specific manually specified version

Since version 2.5.10, the sofie command supports extended syntax for update and upgrade. Specifically:

It is therefore no longer necessary to always move to the latest released version in the branch, but it is possible to specify a particular version to which the update or upgrade should be performed. The version must be higher than the current one (downgrade is not supported) and must exist; otherwise, an error will occur and the action will not be executed.

It still applies that update can only move within the current major version branch, and upgrade within the nearest higher major version branch.

The purpose of this manual version selection is that if, within an organization, an update or upgrade is first performed and tested in a test environment, and only after testing (which may take days or weeks) is it performed in the production environment, a new version may have been released by the vendor during that time. Without specifying a particular version, production would then be updated to a newer and untested version than the one used in the test environment.

Specific instructions for particular versions

Especially when upgrading to a new major version of the application (e.g. from 1.3.x to 1.4.x), it often happens, that the automatic upgrade process cannot perform all the necessary steps and some steps must be done manually (typically modifying the nginx configuration).

Furthermore, when introducing new features or changes, a conservative approach is usually chosen, so that whenever possible, the upgrade does not introduce changes in the behavior of the application. But that also means, that after the upgrade the new features can be disabled and inactive, and to use them, the configuration must be modified.

Version 2.5.9

Necessary manual upgrade steps

No manual steps necessary.

However, official support for increasing the number of concurrently executed file checks has been introduced. Previously, it was supported only unofficially and the default state is set to 1, meaning serial execution.

• We recommend using this if it is necessary to increase file scanning performance and sufficient processors are available.

• It can be set using the command “sofie file-check-concurrency N", where N is the number of concurrently running checks.

• We recommend setting it to a maximum of the number of processors (cores) minus 2. For example, set 2 for 4 cores, set 6 for 8 cores, etc.

• After increasing the value, it cannot be easily reduced again, see: https://wikisonpo.atlassian.net/wiki/spaces/SPEN/pages/4374200321.

Recommended configuration modifications and checks

Apart from the parallelism mentioned above, no changes were made in this version that would require action from the administrator.

Version 2.5.8

Necessary manual upgrade steps

No manual steps necessary.

Recommended configuration modifications and checks

The version 2.5.8 contains new or modified features. The application administrator should, after performing the update, go through the following settings and consider if and how he wants to use these features:

• Settings → Configuration → Packages → File Expiration:

  • A new configuration item “Expiration of shredded files” has been added. It can also be set for individual packages.

  • The structure of the remaining items in this menu has been adjusted.

• View settings for the file list in the package detail (for users and administrators):

  • The new default state is not to display deleted and shredded files.

  • This can be changed using the gear icon in the top right above the file list, where they can be shown again.

  • It is now also possible to select which columns are displayed (some can be hidden).

  • (The view configuration applies individually to each user, not globally. This has not changed.)

Version 2.5.5

Necessary manual upgrade steps

No manual steps necessary.

Recommended configuration modifications and checks

The version 2.5.5 contains new or modified features. The application administrator should, after performing the update, go through the following settings and consider if and how he wants to use these features:

• Settings → Configuration → Security → Password rules → Password rules for packages → Mandatory access link password:

  • A new setting item explicitly determining whether setting a password is required for access links (package sharing).

  • Previously (from 2.5.0 to 2.5.4), this was governed together by the setting “Mandatory package password - logged in user”. After the update, the default will be taken from the value of this setting.

  • The setting does not apply to requests, even though since version 2.5 they are implemented as access links. They have an exception to preserve the behavior from previous versions.

Version 2.5.0

Necessary manual upgrade steps

No manual steps necessary.

For new installations, Certbot is no longer installed for issuing and renewing Let’s Encrypt certificates; instead, the new native nginx functionality is used – the acme module. Existing installations are not migrated and remain as they are. This does not cause any issues.

Recommended configuration modifications and checks

The version 2.5.0 contains new or modified features. The application administrator should, after performing the update, go through the following settings and consider if and how he wants to use these features:

• Settings → User Management:

  • A new menu sub-section consolidating the management of users, groups, administrators, etc. Some sections were moved here from other parts of the settings.

  • Administrators: it is now possible to have remote administrators authenticated via ADFS or OIDC.

  • User groups: new permissions for users:

    • Reset password

    • Share packages

      • Share packages – publicly

      • Share packages – internally

      • Share packages – privately

    • Add files to sent packages

    • Delete, rename, and move files in sent packages

    • Add and manage package files in quarantine

    • Delete files in packages in quarantine

    • Access to the audit log

      • Access to the extended audit log

  • API tokens:

    • It is now possible to reissue an existing token. The original token is invalidated and a new one is displayed. Previously, it was necessary to delete the existing token and create a new one, including all required settings.

    • For new administrative tokens, it is now possible to configure whether they allow impersonation of users. Previously, this was always enabled and could not be disabled.

  • ADFS + OpenID Connect: it is now possible to create integrations for administrator login, not only for users.

• Settings → Configuration → Packages → General:

  • Allow adding of files to a sent package + Allow deleting, renaming and moving of files in a sent package:

    • Changed from a switch (enabled/disabled) to an option Yes/No/Selected only, where the Selected only option allows flexible configuration using permissions.

  • Users can share packages: a new setting determining whether users can share access to packages using new unique access links. Even if this setting is disabled, users can still create requests as before (although they are now implemented using access links).

• Settings → Configuration → Packages → Expiration → Other packages: the setting “Package requests expiration” was removed, because requests were replaced with access links. The validity of the link replacing a request is not limited and is therefore the same as the validity of the given package.

• Settings → Configuration → Packages → File expiration:

  • A new section allowing configuration of whether individual files in packages should expire independently of the package expiration (i.e., earlier than the package itself, suitable for persistent or long-term packages).

• Settings → Configuration → Packages → Size limits: previously, this setting applied only when creating a new package, not to subsequent modifications. It is now always enforced. This may affect users’ work with packages, and it may be appropriate to consider increasing the limit or refining it for different users or groups using the new policies.

• Settings → Configuration → Packages → Access control – new section Access to package data in quarantine, including new options:

  • Users can add and manage package files in quarantine

  • Users can delete files in packages in quarantine

• Settings → Configuration → Users → Multifactor authentication → new option: TOTP time delay [s] – tolerance in seconds during which a just-expired code is still considered valid.

• Settings → Configuration → Logging → new section: Limit logging of unauthorized access attempts. Allows limiting the number of logs of this type to prevent log flooding.

• Settings → Configuration → E-mail – new options:

  • Send e-mails – allows completely disabling sending of all e-mails from the application.

  • Set Reply-To header based on the sender – allows notification e-mails sent to recipients, when a user creates a package or a request, to be configured so that replies go to the user’s e-mail address instead of the global application e-mail address.

• Settings → Configuration → Diagnostics – new options:

  • Send environment information

  • Send error reports

    • Anonymize data

  • By default, all these options are enabled, allowing better diagnostics of potential application issues. We recommend keeping them enabled, or possibly considering disabling data anonymization for even more accurate error analysis.

• Settings → Templates:

  • The default texts of some templates were updated to better reflect the new package options (persistent, cooperative, empty, etc.).

  • New template “PACKAGE_SHARED”.

• Settings → Detection settings:

  • For individual detection engines, so-called “profiles” were introduced, in which their configuration is now stored. Each engine can have multiple detection profiles with different settings. These can then be used in new detection engine policies.

  • After upgrading to the new version, each engine has one profile created, “Default”, to which the entire original configuration is moved. This profile is set as default, so it is used if no other profile is specified in a policy. As a result, there is no change in behavior unless the administrator starts using policies and profiles.

  • The list of engines has a slightly updated design to make it clearer and more readable.

• Settings → Network objects:

  • A new settings section where it is possible to prepare a registry of network objects that can then be used in some parts of the configuration (e.g., policies and ADFS and OIDC). It will likely be further expanded in the future.

  • It is possible to create and name network addresses and ranges in CIDR format, for both IPv4 and IPv6. These addresses can also be grouped.

• Settings → Policies – new section containing policy lists for:

  • Size limits

  • Detection engines

  • Packages expiration

  • Each policy list allows administrators to flexibly define different settings for different users or user groups, or IP addresses and ranges. Policies are evaluated from top to bottom, and the settings from the first matching policy are applied. If no policies are created, the settings from the Default policy are used, which is always fixed and derived from the global application configuration, meaning the behavior is the same as before the introduction of policies.

• Settings → Audit Log + also Settings → Configuration → Logging → Syslog: now includes a ? icon. When clicked, it displays a list of all audit log types valid in the current version, including severity, application version (when introduced), and an example message when it occurs. This inline documentation replaces the manually maintained audit log documentation previously available here in the wiki.

Version 2.4.1

Necessary manual upgrade steps

No manual steps necessary, however, to ensure proper operation of the new "Load Public Key" button in Settings → ADFS, it is recommended to update the nginx configuration file "/etc/nginx/sofie.d/http_common-variables.conf" by adding the specific FQDN (Fully Qualified Domain Name) of the ADFS server. See the example below:

Recommended configuration modifications and checks

The version 2.4.1 contains new or modified features. The application administrator should, after performing the update, go through the following settings and consider if and how he wants to use these features:

• Settings → Configuration → Users → Appearance:

  • User name display. It is now possible to configure how the user’s name is displayed in the top bar after login. Instead of the login (username), users can choose to display either their email address or their full name (first and last name). Each user can also customize this setting in their user profile.

• Settings → ADFS:

  • Title: Optional custom text for the login button for users using ADFS.

Version 2.4.0

Necessary manual upgrade steps

No manual steps necessary.

However, the new Java 21 will be automatically installed, and the system's default Java version will be switched from 1.8 to 21. The application should ideally be installed on a separate dedicated server, as recommended, so this change should not affect anything. If, contrary to recommendations, other services or applications are installed and running on the server, it is necessary to verify that they still function correctly and, if needed, fix them. The old Java 1.8 remains in the system but is no longer the default. If it is not required, it can be removed.

Optionally it is now also possible to restrict access to the admin interface (/admin) not just on the application level, but also on the nginx web server level. It can be done by modifying the configuration file “/etc/nginx/sofie.d/http_admin-restriction.conf”:

Recommended configuration modifications and checks

The version 2.4 contains new or modified features. The application administrator should, after performing the upgrade, go through the following settings and consider if and how he wants to use these features:

• Settings → User Groups:

  • Newly introduced group support. Users can now be grouped either locally within the application or by importing group memberships through mapping from an external directory (AD/ADFS/OIDC).

  • To enable group mapping from ADFS, it is necessary to add an attribute to the claims. Refer to the guide: https://wikisonpo.atlassian.net/wiki/spaces/SPEN/pages/2191196210.

  • Permissions moved from users to groups. All user permissions have been automatically migrated during the upgrade to corresponding newly created groups, of which the users are members.

  • Default permissions for new users (previously in Settings → Configuration → Users) are now moved to the default group, "default".

• Settings → API Tokens:

  • In addition to user tokens, administrator tokens are now supported. The API has been substantially extended. See the documentation: https://docs.sofie.cloud/api/v2/en/ .

• Settings → Configuration → Users:

  • A new option, User Login - Allow direct user login, has been introduced.

• Settings → Configuration → SFTP Server:

  • New section and functionality. This feature requires an additional license.

• Settings → Detection Settings:

  • Internal Modules → MIME types detection, new options:

    • Maximum embedded content depth (e.g. for archives and documents)

    • Limit maximum number of embedded objects

      • Maximum number of embedded objects

  • Sandboxes:

    • A new option, “Check only selected file types”, has been added for all sandboxes. This allows limiting inspections to specified file types only (for example those that the sandbox can process).

Version 2.3.11

Necessary manual upgrade steps

If we use the new (since version 2.3.0) “Content Disarm and Reconstruction (CDR)“ module, it needs modifications for optimal function, performed by running the following commands in the command line:

Version 2.3.0

Necessary manual upgrade steps

If we want to use the new “Content Disarm and Reconstruction (CDR)“ module, it needs to be installed first according to the following instructions: https://wikisonpo.atlassian.net/wiki/spaces/SPEN/pages/899153928/Installation+manual#Installation-of-the-CDR-module-(optional)

Recommended configuration modifications and checks

The version 2.3 contains new or modified features. The application administrator should, after performing the upgrade, go through the following settings and consider if and how he wants to use these features:

• Settings → API Tokens → for individual tokens:

  • Allowed IP ranges

• Settings → Configuration → Security:

  • new section: Access Restrictions, including:

    • Access Restrictions - IP Filter

    • Access restrictions - reverse record lookup

    • Access restrictions - country

    • Access restrictions - User-Agent header

• Settings → Configuration → Data Encryption:

  • Key encryption keys (KEK)

    • new recommended key type: HPKE

• Settings → Configuration → Users:

  • Default settings for new users, new permissions:

    • Set package as persistent

    • Persistent account

• Settings → Detection Settings:

  • CDR (Content Disarm and Reconstruction)

    • new integrated engine for converting compatible content into a safe form

  • Avast, Trellix

    • newly supported antiviruses

• Settings → ADFS:

  • new attribute: Directory (tenant) ID

    • no need to change after upgrade to work as before, for ADFS it should be “adfs”

  • newly displayed: Redirect URI

• Settings → OpenID Connect:

  • an entirely new section allowing integration with OIDC user sources and logins (including Azure AD or Google)

Version 2.2.0

Necessary manual upgrade steps

No manual steps necessary.

Recommended configuration modifications and checks

The version 2.2 contains new or modified features. Also the structure of the Settings → Configuration menu has undergone serious overhaul. The application administrator should, after performing the upgrade, go through the following settings and consider if and how he wants to use these features:

• Settings → Users → for individual users:

  • User’s email aliases

  • Set approvers

• Settings → Configuration → Basic:

  • Configuration management - Configuration export

• Settings → Configuration → Packages → General:

  • Users can use the briefcase

  • Users can create cooperative packages

• Settings → Configuration → Packages → Expiration:

  • Shredded packages

    • Automatically delete metadata about shredded packages

    • Expiration of shredded packages' metadata

• Settings → Configuration → Packages → Access control:

  • Closed environment

    • IP ranges of the closed environment

    • Allow closed → closed transfer

    • Allow closed → open transfer

    • Allow open → closed transfer

    • Allow open → open transfer

    • Allow upload of packages from the closed environment

    • Allow upload of packages from an open environment

• Settings → Configuration → Packages → Approval of sending:

  • Preferred approver

  • A user can choose his approvers

  • An approver can access all packages that require approval

• Settings → Configuration → Users:

  • Default settings for new users, new permissions:

    • Use the briefcase

    • Access all packages that require approval

    • Select the approver of packages to be sent

    • Access to detailed check results

    • Create cooperative packages

  • Access detailed results of file checks

    • Logged in users

    • Anonymous users

• Settings → Configuration → Logging:

  • Retention of audit logs

    • Automatically delete old audit logs

    • Delete audit logs older than

• Settings → Templates:

  • Modified some existing default templates.

• Settings → Detection settings:

  • MIME

    • Maximum number of stored records of detected objects

Version 2.1.0

Necessary manual upgrade steps

Since this version the structure of the nginx configuration was substantially modified and also the update and upgrade process was improved in such a way, that in most cases no manual intervention should be necessary. For this improved automation to work correctly, it is required that the installation adheres to the instructions from the installation manual, including:

• The installation should be done on a dedicated server only for this purpose. Especially it should not contain any manual modifications to the nginx configuration. If it does contain such manual modifications, those will be overwritten during the upgrade to the version 2.1 and the administrator will have to manually fix it or do again.

• If a custom own SSL/TLS certificate for HTTPS is used, it is expected it will be in the default path mentioned in the installation manual. If another path for the key/certificate was used, the same thing as above in the case of custom nginx modifications applies (will be overwritten and will need manual fix).

The upgrade specifically overwrites the files /etc/nginx/conf.d/default.conf and /etc/nginx/nginx.conf. But it creates a backup of the original files before overwriting them, which look like this:

Recommended configuration modifications and checks

The version 2.1 contains new or modified features. The application administrator should, after performing the upgrade, go through the following settings and consider if and how he wants to enable and use the new features:

• Settings → Configuration → Basic settings:

  • Allowed languages for administrators

  • Allowed languages for users

• Settings → Configuration → Security:

  • new section: Password rules for packages, including:

    • Mandatory package password - not logged in user

    • Mandatory package password - logged in user

  • Login and authorization lifetime:

    • Validity period of one access to a package with a limited number of accesses

• new section: Settings → Configuration → Security limits, including:

  • section Limit authentication attempts

  • section Limit password reset requests

  • section Limit attempts to access a password protected package

  • section Limit attempts to access a password protected request

  • section Limit attempts to access a non-existent package

  • section Packages with a limited number of accesses

• Settings → Configuration → Notifications:

  • new section: Approval of sent packages

  • System notifications

    • Notify when the detection engine's quota is exceeded

• Settings → Configuration → Appearance:

  • new section: Display the terms of use

• Settings → Configuration → User settings:

  • Default settings for new users, new permissions:

    • Do not enforce a multifactor authentication

    • Approve sending of packages

    • Send packages without additional approval

    • Send packages to yourself

    • Download files from own sent packages

  • new section: Multifactor authentication:

    • Require multifactor authentication for local users

    • Require multifactor authentication for AD users

    • Require multifactor authentication for ADFS users

  • new section: User restrictions by IP ranges

  • new section: Settings for sent packages, including:

    • Users can send packages to themselves

    • Users can download files from their own sent packages

    • Users can send packages without approval

    • Allow access to packages being approved without requiring their password

• Settings → E-mail templates renamed to Templates:

  • Modified some existing default templates.

  • Created new templates: DETECTION_ENGINE_QUOTA_EXCEEDED, EMAIL_TEMPLATE_CSS, EMAIL_TEMPLATE_PREFIX, EMAIL_TEMPLATE_SUFFIX, PACKAGE_APPROVE_REQUEST, SEND_PACKAGE_APPROVED, SEND_PACKAGE_DISAPPROVED, TERMS_OF_USE.

Version 2.0.3

Necessary manual upgrade steps

• Edit the file /etc/nginx/proxy.conf and add the following line to the end:

  proxy_buffering off;

After this change it is necessary to restart or reload nginx:

Version 2.0.0

Necessary manual upgrade steps

Particularly because of the changed behavior, where the downloaded files are no longer served directly by nginx, but by the SOFiE application itself, the nginx configuration must be changed in the following way:

• Delete the files:

  • /etc/nginx/download-auth.conf

  • /etc/nginx/download.conf

• Modify the file: /etc/nginx/conf.d/default.conf

  • Delete the sections:

    location /api/ { client_max_body_size 0; proxy_pass http://127.0.0.1:9090/api/; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_request_buffering off; } location ~/admin/download-file/(.+) { include download.conf; auth_request /download-file-auth-admin; } location ~/admin/download-archive/(.+) { include download.conf; auth_request /download-file-auth-admin; } location ~/download-file/(.+) { include download.conf; auth_request /download-file-auth-user; } location ~/download-archive/(.+) { include download.conf; auth_request /download-file-auth-user; } location /download-file-auth-user { include download-auth.conf; proxy_pass http://127.0.0.1:9090/api/user$request_uri; } location /download-file-auth-admin { include download-auth.conf; proxy_pass http://127.0.0.1:9090/api$request_uri; }

  • In their place add the following new sections:

    add_header X-Content-Type-Options "nosniff"; add_header Referrer-Policy "same-origin"; location /api/ { set $x_uri_suffix ""; include proxy.conf; } location /admin/download-file/ { set $x_uri_suffix "/api"; include proxy.conf; } location /admin/download-archive/ { set $x_uri_suffix "/api"; include proxy.conf; } location /download-file/ { set $x_uri_suffix "/api/user"; include proxy.conf; } location /download-archive/ { set $x_uri_suffix "/api/user"; include proxy.conf; }

• Create a new file: /etc/nginx/proxy.conf with the following contents:

  client_max_body_size 0; proxy_pass http://127.0.0.1:9090$x_uri_suffix$request_uri; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_request_buffering off;

After those changes it is necessary to restart or reload nginx:

Recommended configuration modifications and checks

The version 2.0 contains a lot of new or modified features. The application administrator should, after performing the upgrade, go through the following settings and consider if and how he wants to enable and use the new features:

• Settings → Configuration → Basic settings: