Retention Policy: The clean up job queue
Hello there đź‘‹
Data is essential to any ERP system, unless the data is irrelevant or so old that it’s no longer needed (depending on your local policies). This is true whether you archive some records or create log entries for a feature of your extension. If these records would also store files in a blob field aswell, chances are that sooner or later you will run out of storage space.
As of Version 17.x (that’s already two years ago) Microsoft introduced the possibility to Define retention policies1. Pretty big deal as this is a big productivity boost for developers and a good way for customers to keep control of the data (e.g. how long the data would be allowed to stay in the system, by defining a Retention Period).
If there is a risk of data loss, it is advisable to contact the extension publisher. Conclusion: only allow tables where the data is not important at all.
You can find the Retention Policies page by searching for retention via Tell Me:
After opening the Retention Policies page you will be displayed with the already created records. The list also gives you a quick overview of which records are enabled and which Retention Period is set for the individual entry:
As this list is company specific, it’s important to make use of an event that is triggered either after login or after initialization of a new company. This way we ensure that the tables are allowed for newly created companies. Check out my last blog post) for examples on Company-Initialize.
Without further ado, let’s take a look at an example:
codeunit 50100 "Module Installer"
{
Subtype = Install;
Access = Internal;
Permissions = tabledata Field = r;
trigger OnInstallAppPerCompany()
begin
AddRetentionPolicyAllowedTables();
end;
procedure AddRetentionPolicyAllowedTables()
begin
AddRetentionPolicyAllowedTables(false);
end;
procedure AddRetentionPolicyAllowedTables(ForceUpdate: Boolean)
var
Field: Record Field;
UpgradeTag: Codeunit "Upgrade Tag";
RetenPolAllowedTables: Codeunit "Reten. Pol. Allowed Tables";
IsInitialSetup: Boolean;
begin
IsInitialSetup := not UpgradeTag.HasUpgradeTag(GetEmailTablesAddedToAllowedListUpgradeTag());
if not (IsInitialSetup or ForceUpdate) then
exit;
RetenPolAllowedTables.AddAllowedTable(Database::"Demo Log");
if IsInitialSetup then
UpgradeTag.SetUpgradeTag(GetEmailTablesAddedToAllowedListUpgradeTag());
end;
local procedure GetEmailTablesAddedToAllowedListUpgradeTag(): Code[250]
begin
exit('CB-1234-DemoLogEntryAdded-20221217');
end;
[EventSubscriber(ObjectType::Codeunit, Codeunit::"Reten. Pol. Allowed Tables", 'OnRefreshAllowedTables', '', false, false)]
local procedure AddAllowedTablesOnRefreshAllowedTables()
begin
AddRetentionPolicyAllowedTables(true);
end;
[EventSubscriber(ObjectType::Codeunit, Codeunit::"System Initialization", 'OnAfterLogin', '', false, false)]
local procedure AddAllowedTablesOnAfterSystemInitialization()
begin
AddRetentionPolicyAllowedTables();
end;
}
The code snippet above is a slightly modified copy of the 1596 “Email Installer” codeunit. There is a lot of overhead in this codeunit, I agree. The only relevant line for now is RetenPolAllowedTables.AddAllowedTable(Database::"Demo Log");. This will add our custom Demo Log table to the allowed tables that can be picked up by the Retention Policy. Only now we are able to add a new Retention Policy based on that table.
This simplified example is probably not sufficient because you only want to refer to specific records in this table. That’s where the procedure AddTableFilterToJsonArray comes in. But first take a look at the AddAllowedTable procedures from the 3905 “Reten. Pol. Allowed Tables” codeunit. Thanks to overloading we are looking for the procedure AddAllowedTable(TableId: Integer; TableFilters: JsonArray): Boolean. If you want to set the mandatory minimum retention days and still be able to filter you have to provide a few more parameters to the procedure AddAllowedTable(TableId: Integer; DefaultDateFieldNo: Integer; MandatoryMinRetenDays: Integer; RetenPolFiltering: Enum "Reten. Pol. Filtering"; RetenPolDeleting: Enum "Reten. Pol. Deleting"; TableFilters: JsonArray): Boolean.
The official documentation of the Define Data Retention Policies2 provides the following example which includes a mandatory filter:
trigger OnInstallAppPerCompany()
var
ChangeLogEntry: Record "Change Log Entry";
RetenPolAllowedTables: Codeunit "Reten. Pol. Allowed Tables";
RetentionPeriod: Enum "Retention Period Enum";
RecRef: RecordRef;
TableFilters: JsonArray;
Enabled: Boolean;
Mandatory: Boolean;
begin
ChangeLogEntry.Reset();
ChangeLogEntry.SetFilter("Field Log Entry Feature", '%1|%2', ChangeLogEntry."Field Log Entry Feature"::"Monitor Sensitive Fields", ChangeLogEntry."Field Log Entry Feature"::All);
RecRef.GetTable(ChangeLogEntry);
Enabled := true;
Mandatory := true;
RetenPolAllowedTables.AddTableFilterToJsonArray(TableFilters, RetentionPeriod::"28 Days", ChangeLogEntry.FieldNo(SystemCreatedAt), Enabled, Mandatory, RecRef);
RetenPolAllowedTables.AddAllowedTable(Database::"Change Log Entry", ChangeLogEntry.FieldNo(SystemCreatedAt), TableFilters);
end;
At this point I would like to recommend to you to check out the blog post of Stefano Demiliani - Dynamics 365 Business Central: creating retention policies from AL3 and the blog post of Krzysztof Bialowas - Clean after yourself – a few words about Retention Policies4. These MVPs describe the feature in more depth.
If you are curious what’s inside the Retention Policy JQ you should check out the amazing repository5 provided by Stefan Maron.
Sometimes users are responsible and want to perform the action manually. In this case, the operation may fail due to insufficient permissions. Let’s have another look at the 1598 “Email Logs Delete” codeunit.
What we can see is a property Permissions on top of the object and an [EventSubscriber(ObjectType::Codeunit, Codeunit::"Apply Retention Policy", 'OnApplyRetentionPolicyIndirectPermissionRequired'.
...
Permissions = tabledata "Demo Log" = rd;
...
[EventSubscriber(ObjectType::Codeunit, Codeunit::"Apply Retention Policy", 'OnApplyRetentionPolicyIndirectPermissionRequired', '', true, true)]
local procedure DeleteRecordsWithIndirectPermissionsOnApplyRetentionPolicyIndirectPermissionRequired(var RecRef: RecordRef; var Handled: Boolean)
var
RetentionPolicyLog: Codeunit "Retention Policy Log";
begin
if Handled then
exit;
if not (RecRef.Number in [Database::"Demo Log"]) then
exit;
// if no filters have been set, something is wrong.
if (RecRef.GetFilters() = '') and (not RecRef.MarkedOnly()) then
RetentionPolicyLog.LogError(Enum::"Retention Policy Log Category"::"Retention Policy - Apply", StrSubstNo(NoFiltersErr, RecRef.Number, RecRef.Name));
RecRef.DeleteAll(true);
Handled := true;
end;
...
Notice that we create an error in the Retention Policy Log whenever no filters have been set.
This was a brief overview of the Retention Policy feature in Dynamics 365 Business Central.
Hope you learned something new.
See you around 🦦
-
https://learn.microsoft.com/en-us/dynamics365-release-plan/2020wave2/smb/dynamics365-business-central/planned-features#application ↩
-
https://learn.microsoft.com/en-us/dynamics365/business-central/admin-data-retention-policies ↩
-
https://demiliani.com/2020/11/18/dynamics-365-business-central-creating-retention-policies-from-al/ ↩
-
https://www.mynavblog.com/2020/12/21/clean-after-yourself-a-few-words-about-retention-policies/ ↩